Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
standard rename rows (M-088/M-089) under the spec's missed-rename clause.
Docs + ledger + test-count updates only - **no public API or numerical behavior
change.**
- **MMM calibration export (`diff_diff.mmm`)** - interop builders that assemble
Marketing Mix Model calibration inputs from experiment results, with no MMM
package dependency and no result introspection (purely additive). Design is
explicit-in / validated-out: the caller supplies the already-scoped
incremental outcome and its SE (aggregated to the population and window one
MMM row represents), and diff-diff assembles the schema, enforces each
consumer's guards, and converts to the target parameterization - it does not
rescale a headline ATT, because that reconciliation needs the MMM's row
structure and outcome scale it cannot see.
`to_pymc_marketing_lift_test(channel, x, delta_x, delta_y, sigma, dims=,
on_wrong_sign=)` builds the PyMC-Marketing/prophetverse lift-test DataFrame
with sign/zero (Gamma-likelihood)/positivity guards.
`to_meridian_roi_prior(incremental_outcome, incremental_outcome_se, spend,
parameter="roi_m"|"mroi_m", se_widening=)` builds Google Meridian lognormal
ROI/mROI priors (`MeridianROIPrior` with `mu`/`sigma` matching Meridian's
`lognormal_dist_from_mean_std`, spend-weighted multi-experiment pooling, and a
channel- and time-scoped `.to_code()` snippet that sets `media_prior_type`).
Deriving totals directly from a fitted result is deferred to the post-4.0
`results.aggregate()` layer.

- **Reviewer-eval harness: N-arm matrix, blinded grading, corpus grown 2 -> 11
(`tools/reviewer-eval/`, prep for the GPT-5.6 reviewer evaluation).**
`config/configs.json` moves from the two-arm `control`/`candidate` shape to an
Expand Down
1 change: 1 addition & 0 deletions DEFERRED.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ For survey-specific limitations (`NotImplementedError` paths), see the

| Issue | Location | PR | Priority |
|-------|----------|----|----------|
| MMM interop: result-derived scaling for the exporters - let a fitted result produce the scoped incremental outcome + SE directly (via the post-4.0 `results.aggregate()` layer, where the estimator owns its aggregation weights/balance/survey masses), so callers need not hand-scale the ATT. v1 is explicit-in by design; this is the seamless follow-up. | `diff_diff/mmm.py`, `docs/v4-design.md` | mmm-interop | Low |
| `SyntheticControl` fit-snapshot residency (`_SyntheticControlFitSnapshot`) — **investigated 2026-07-07, parked**: the snapshot ALIASES the fit's own working pivots (zero extra construction cost); the retained residency implements the documented freeze contract (post-fit mutation of estimator inputs must not change `in_space_placebo()` / `leave_one_out()` / conformal output on an already-returned results object, and `__getstate__` already excludes it from pickles). A compact array representation saves only pandas overhead (the float panel dominates); releasing residency needs new API surface (`release`/opt-out flag) or a freeze-contract change. Revisit on user demand for very large donor panels. | `synthetic_control.py`, `synthetic_control_results.py` | follow-up | Low |
| Stratified survey-PSU multiplier-weight draw-tiling — **investigated 2026-07-07, parked**: the stratified generator (`generate_survey_multiplier_weights_batch`) consumes ONE sequential rng stream stratum-major (`rng.choice(size=(n_bootstrap, n_h))` per stratum, then lonely-PSU pooling), so draw-chunked assembly CANNOT reproduce the stream bit-identically (contra the old row's parenthetical) — it would need per-stratum generator state skipping (PCG64.advance + per-weight-type variate accounting; fragile) or a stream-layout change (MC-level SE changes → baseline/golden recapture + REGISTRY note). Stratified designs have few PSUs, so the full `(n_bootstrap × n_psu)` matrix rarely matters; unstratified (the large-`n_units` case) is already tiled. Revisit only if a large-PSU stratified design hits memory, as a documented stream change. | `diff_diff/bootstrap_chunking.py::iter_survey_multiplier_weight_blocks` | follow-up | Low |
| ChangesInChanges FULL Melly-Santangelo covariate estimator (monotonized integrated-indicator conditional CDFs, treated-post `F_{X|11}` integration, exchangeable bootstrap with variance-weighted KS bands, tail trimming, pre-period specification test). The qte-`xformla` simplified form of the MS pipeline SHIPPED in the covariates PR (`covariates=` on both estimators, parity-tested vs qte 1.3.1); this row's earlier "No R parity target exists" claim was WRONG and is corrected in the MS review doc. The full estimator has no reference implementation (the MS Stata code is the only one; distinct from Kranker's `cic`) and would need simulation-based validation. Reviewed: `docs/methodology/papers/melly-santangelo-2015-review.md`. | `diff_diff/changes_in_changes.py` | #682 | Low |
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ Full guide: `diff_diff.get_llm_guide("practitioner")`.
- [Honest DiD](https://diff-diff.readthedocs.io/en/stable/api/honest_did.html) - Rambachan & Roth (2023) sensitivity analysis: robust CI under PT violations, breakdown values
- [Pre-Trends Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/pretrends.html) - Roth (2022) minimum detectable violation and power curves
- [Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/power.html) - analytical and simulation-based MDE, sample size, power curves for study design
- [MMM Calibration Export](https://diff-diff.readthedocs.io/en/stable/api/mmm.html) - convert experiment results into MMM calibration inputs: PyMC-Marketing lift-test frames and Google Meridian lognormal ROI priors
- Conley spatial HAC SE (`vcov_type="conley"`) on cross-sectional `LinearRegression` / `compute_robust_vcov` plus panel `DifferenceInDifferences` / `MultiPeriodDiD` / `TwoWayFixedEffects` (with `conley_lag_cutoff` for within-unit Bartlett temporal HAC) - Conley (1999) spatial-correlation-aware SEs with parity vs R `conleyreg` on cross-sectional + panel fixtures, optional combined spatial + cluster product kernel via explicit `cluster=`, auto-activating sparse k-d-tree fast path for `n > 5_000`

## Survey Support
Expand Down
2 changes: 2 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,5 +43,7 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m
|-------|----------|--------|--------|----------|
| Align the four legacy dataset loaders (`load_card_krueger`, `load_castle_doctrine`, `load_divorce_laws`, `load_mpdta`) with the loud-fallback pattern of `load_prop99`/`load_walmart`: `UserWarning` + `df.attrs["source"]` marker on synthetic fallback (currently silent), plus optional checksum pinning for the CSV downloads. **Upgraded to a live defect 2026-07-13: the `causaldata/causal_datasets` GitHub repo backing castle/card_krueger/divorce is dead (404), so those loaders silently serve synthetic data everywhere - needs loud fallback + replacement sources.** | `diff_diff/datasets.py` | LWDiD precursor | Quick | Medium |
| Tighten the mypy suppressions that back the enforced-zero posture: burn down `prep_dgp`'s per-module `[index]` override (needs a None-vs-array restructure that preserves the seeded RNG stream), and evaluate re-enabling the globally disabled codes (`arg-type`, `return-value`, `var-annotated`, `assignment`) one at a time — `assignment` alone hid several real annotation drifts found during the 2026-07 triage. | `pyproject.toml` `[tool.mypy]`, `diff_diff/prep_dgp.py` | lint-CI | Mid | Low |
| MMM interop follow-up: Meridian `roi_calibration_period` mask builder - accept the MMM's time index + channel order and emit the boolean `(n_media_times, n_media_channels)` mask so `.to_code()` scopes the prior to the experiment window automatically (today the caller passes a mask expression / `full_model_window=True`). | `diff_diff/mmm.py` | mmm-interop | Quick | Low |
| MMM interop PR-B: calibration tutorial notebook (fit DiD/CS -> scope -> `to_pymc_marketing_lift_test` / `to_meridian_roi_prior`) + a `llms-practitioner.txt` Step 8 pointer to the exporters as the MMM hand-off. | `docs/tutorials/`, `diff_diff/guides/llms-practitioner.txt` | mmm-interop | Mid | Low |
| Tracking-file contract guard test: reject NEW active deferred-work pointers at `TODO.md` (deferred rows live in `DEFERRED.md`; allowlist for historical/past-tense prose and actionable-row pointers) and assert rows cross-linking a `docs/v4-deprecations.yaml` `M-xxx` id don't restate ledger status. Origin: tracking-split local review R2. | `tests/`, `TODO.md`, `DEFERRED.md` | tracking-split | Quick | Low |
| Real-data CI canary for dataset-backed replication tests: `test_methodology_lwdid.py`'s Prop 99 / Walmart goldens skip (visibly) when loaders fall back to synthetic; add a lane or canary asserting `df.attrs["source"] == "lwdid_ssc_ancillary"` in CI so network regressions cannot silently de-gate the replication tests. Pairs with the loader-fallback repair row above. | `tests/test_methodology_lwdid.py`, `.github/workflows/` | LWDiD validation suite | Quick | Low |
9 changes: 9 additions & 0 deletions diff_diff/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,11 @@
)
from diff_diff.lpdid import LPDiD
from diff_diff.lpdid_results import LPDiDResults
from diff_diff.mmm import (
MeridianROIPrior,
to_meridian_roi_prior,
to_pymc_marketing_lift_test,
)
from diff_diff.power import (
PowerAnalysis,
PowerResults,
Expand Down Expand Up @@ -566,6 +571,10 @@
"Alert",
"OutcomeShape",
"TreatmentDoseShape",
# MMM calibration export (interop)
"to_pymc_marketing_lift_test",
"to_meridian_roi_prior",
"MeridianROIPrior",
# LLM guide accessor
"get_llm_guide",
]
Expand Down
83 changes: 82 additions & 1 deletion diff_diff/guides/llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2042,7 +2042,6 @@ sample_result = compute_sample_size(effect_size=0.5, sigma=1.0)

# Simulation-based power
sim_result = simulate_power(
n_units=200, n_periods=8, treatment_period=4,
effect_sizes=[0.1, 0.5, 1.0, 2.0],
n_simulations=500, seed=42,
)
Expand Down Expand Up @@ -2731,3 +2730,85 @@ cross-period aggregations are enumerated in
current release; see that document for phrasing rules, the
no-traffic-light decision, unit-translation policy, and schema
stability policy.

## MMM Calibration Export

Interop builders converting experiment results into Marketing Mix Model
(MMM) calibration inputs. Pure numpy/pandas - no MMM package is imported,
no result object is introspected (the module is purely additive). Design:
EXPLICIT IN, VALIDATED OUT. Reconciling an experiment estimate to a
calibration input needs the target MMM's row granularity (per-geo vs
national), its time window, and the outcome scale (additive levels vs
log/rate/share) - none of which diff-diff can see - so the CALLER supplies
the already-scoped incremental outcome and its SE (read off summary(),
aggregated to the population/window one MMM row represents). diff-diff
assembles the schema, enforces each consumer's guards, converts to the
lognormal parameterization, pools, and emits snippets. Deriving totals
from a fit is deferred to the post-4.0 results.aggregate() layer, where
the estimator owns its aggregation weights/balance/survey masses.

```python
from diff_diff import (
SyntheticDiD,
to_pymc_marketing_lift_test,
to_meridian_roi_prior,
)

# Single-treated-geo experiment: with exactly ONE treated geo the SDID ATT
# is that geo's lift, so a geo-labelled row is sound. For MULTIPLE treated
# geos the pooled ATT is an average - export each geo's own effect or omit
# the geo dim and match an aggregate lift to aggregate spend.
result = SyntheticDiD().fit(panel, outcome="revenue", treatment="treated",
unit="geo", time="week")

# PyMC-Marketing / prophetverse lift-test frame. Pass the scoped effect.
df_lift = to_pymc_marketing_lift_test(
channel="tv", # must match the MMM's channel_columns
x=50_000.0, # baseline spend for this row's scope
delta_x=20_000.0, # spend change (nonzero; negative go-dark,
# with x + delta_x >= 0)
delta_y=result.att, # measured lift, scoped to this row
sigma=result.se, # its SE (finite, > 0)
dims={"geo": "US-CA"}, # optional model-dim coordinate (one treated geo)
on_wrong_sign="raise", # raise|drop|keep - PyMC rejects
# sign(delta_y)!=sign(delta_x) AND delta_y==0
# (degenerate for its Gamma lift likelihood)
)

# Google Meridian lognormal prior. Caller aggregates the ATT to a total
# incremental outcome over the treated population/window and its SE.
prior = to_meridian_roi_prior(
incremental_outcome=180_000.0, # total incremental revenue
incremental_outcome_se=45_000.0, # its SE
spend=200_000.0, # channel spend the outcome is over
parameter="roi_m", # roi_m (full-spend/zero-spend return)
# | mroi_m (marginal return)
se_widening=1.5, # >=1 for transferability skepticism
)
prior.roi_mean, prior.roi_sd # pooled ROI moments
prior.mu, prior.sigma # LogNormal params (match Google's
# lognormal_dist_from_mean_std)
prior.to_dict() # JSON-ready
print(prior.to_code( # ready-to-paste PriorDistribution +
channel="tv", # ModelSpec; roi_m/mroi_m is per-channel
media_channels=["search", "tv"], # so channel scope is required (vector
# prior in model channel order; other
# channels keep the Meridian default),
roi_calibration_period="mask", # AND time scope (mask expr) or
# full_model_window=True; sets
)) # media_prior_type accordingly.
```

Key points:
- Lift frame: one row per experiment, columns [channel, *dims, x, delta_x,
delta_y, sigma]; guards on sigma>0, delta_x!=0, x>=0, x+delta_x>=0,
finite delta_y; dims must not collide with reserved columns and must
share one key set. on_wrong_sign governs wrong-sign AND zero-lift rows
(drop-all raises; keep warns the frame is not valid PyMC input).
- Meridian: roi = incremental_outcome/spend, spend-weighted pooling,
roi_sd = sqrt(sum((w_i*sd_i)^2)) (independence assumed - widen via
se_widening); non-positive pooled ROI raises (lognormal positivity);
all scaled/pooled outputs re-validated finite-positive.
- parameter="roi_m" vs "mroi_m" picks the Meridian estimand + its default
for non-experiment channels (LogNormal(0.2,0.9) vs LogNormal(0.0,0.5))
and the emitted media_prior_type ("roi" vs "mroi").
1 change: 1 addition & 0 deletions diff_diff/guides/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ Full practitioner guide: call `diff_diff.get_llm_guide("practitioner")`
- [Honest DiD](https://diff-diff.readthedocs.io/en/stable/api/honest_did.html): Rambachan & Roth (2023) sensitivity analysis — robust CI under parallel trends violations, breakdown values
- [Pre-Trends Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/pretrends.html): Roth (2022) Section II.A-B no-individually-significant (NIS) box-probability pretest power + minimum detectable violation; `pretest_form='nis'` (default) implements the paper's primary form, `pretest_form='wald'` retained as paper-supported alternative (Propositions 1+3+4 all apply); linear-violation MDV in Roth's γ units when relative-time labels are threaded through `fit()`; full Σ_22 routing on non-bootstrap CallawaySantAnna and SunAbraham adapters
- [Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/power.html): Analytical and simulation-based power analysis — MDE, sample size, power curves for study design
- [MMM Calibration Export](https://diff-diff.readthedocs.io/en/stable/api/mmm.html): Assemble Marketing Mix Model calibration inputs from experiment results (explicit-in / validated-out - the caller passes the already-scoped incremental outcome + SE, the module does NOT rescale a headline ATT). `to_pymc_marketing_lift_test(channel, x, delta_x, delta_y, sigma, dims=, on_wrong_sign=)` builds the PyMC-Marketing/prophetverse lift-test DataFrame with sign/zero/positivity guards. `to_meridian_roi_prior(incremental_outcome, incremental_outcome_se, spend, parameter="roi_m"|"mroi_m", se_widening=)` builds Google Meridian lognormal ROI priors (spend-weighted pooling, lognormal parity with `lognormal_dist_from_mean_std`, channel- and time-scoped `.to_code()` snippet setting `media_prior_type`). Pure numpy/pandas; imports no MMM package; does not introspect result objects. Deriving totals from a fit is deferred to the post-4.0 `results.aggregate()` layer.
- Conley spatial HAC SE (`vcov_type="conley"`) on cross-sectional `LinearRegression` / `compute_robust_vcov` PLUS panel `DifferenceInDifferences` / `MultiPeriodDiD` / `TwoWayFixedEffects` (with `conley_lag_cutoff=<int>` for within-unit Bartlett temporal HAC) — Conley (1999) spatial-correlation-aware SEs with haversine/euclidean/callable distance metric and Bartlett/uniform spatial kernel; panel path uses the R `conleyreg`-form block-decomposed sandwich (within-period spatial + within-unit Bartlett serial, same-time excluded); parity vs R `conleyreg` (Düsterhöft 2021) on cross-sectional AND panel `lag_cutoff > 0` fixtures. Combining with explicit `cluster=<col>` applies the combined spatial + cluster product kernel `K_total[i,j] = K_space · 1{c_i = c_j}` (cluster must be constant within each unit across periods on the panel path; validator-enforced). DiD takes `unit=<col>` as a fit-time kwarg when `vcov_type="conley"` (not on `__init__`). Sparse k-d-tree fast path auto-activates for `n > 5_000` with bartlett kernel + haversine/euclidean metric

## Tutorials
Expand Down
Loading
Loading