Add observation-based scoring for filters

[mattlevine22]

Replaces #259 (codex 1 shot re-implementation of #259 using more recent main as starting point).

Trying to get this up in time for @baptistar's summer school lectures at Duke!

Summary

• Adds scoring of one-step-ahead predictive observation distributions for continuous-time KF, EKF, UKF, and EnKF.
• Includes Gaussian log probability, Dawid–Sebastiani, observation-wise CRPS, and energy scores.
• Structures the implementation into:
  • reusable score definitions under dynestyx.evaluation;
  • filter-specific scoring configuration;
  • a canonical predicted-observation representation that translates CD-Dynamax outputs;
  • Filter/ConditionedResult integration and optional NumPyro sites.
• Adds API documentation and Tutorial 12, which compares EKF, UKF, and EnKF using per-time scores and their temporal mean/SD.
• Moves plotting utilities from dynestyx.diagnostics to dynestyx.evaluation.

Reflections

• It would be useful to include a high-particle PF reference, but neither Dynestyx nor CD-Dynamax currently exposes the required pre-update predictive particles and weights. Post-update particles have already incorporated the observation being scored and therefore are not a valid substitute.
• Can we really justify ObservationScoringConfig over directly passing around a Scoring object? Would make hard-coding a scorer easier...but maybe we need a lot of this metadata, so it is better to do the Config style approach?

[DanWaxman]

I find attaching the config to the filter this way a little clunky -- do you have thoughts on something like this? It's what I had settled on when thinking about some diagonistics.

diagnostic_config = DiagnosticConfig(
    n_particles=1_000,
    rng_seed=1337,
    compute_M_diagnostic=True,
    compute_U_diagnostic=True,
    compute_F_diagnostic=True,
    record_log_weights=True,
    record_target_particles=True,
)
with ComputeDiagnostics(diagnostic_config):
    with Filter(filter_config=filter_config):
        dsx.sample("f", dynamics, obs_times=obs_times, obs_values=obs_values)

One reason I find it a bit clunky is that we would have to duplicate some code across different inference context where we want to compute diagnostics -- filtering, smoothing, and LatentPathBuilder.

Another reason to prefer this is then the same object can be configured to automatically do things like "score at lead time" or something.

[mattlevine22]

Mostly agree, but we might want Filter to return different things depending on the diagnostic request.

[DanWaxman]

Sure, that should be possible. Right now, we do a kind of weird thing where handlers and interpretations are co-identified, but this can totally be part of the handler and not part of the interpretation. I.e., the handler records some information about quantities that should be computed, and the interpretation of condition actually computes them.

…

On Thu, Aug 6, 2026 at 4:00 PM Matt Levine ***@***.***> wrote: *mattlevine22* left a comment (BasisResearch/dynestyx#296) <#296 (comment)> Mostly agree, but we might want Filter to return different things depending on the diagnostic request. — Reply to this email directly, view it on GitHub <#296?email_source=notifications&email_token=AC2DPOP5GPNZD37GDR35VWT5ITPUJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRQHA3TINZVGAZ2M4TFMFZW63VQOJSXM2LFO5PXEZLROVSXG5DFMSSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5208747503>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2DPOOBG6EZOX2YP7OOM4L5ITPUJAVCNFSNUABGKJSXA33TNF2G64TZHMYTAOJUGM4TIOBXGE5US43TOVSTWNJQG4ZTSOBYGEYTJILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AC2DPOLSWXLWRS6A2IKCNOT5ITPUJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRQHA3TINZVGAZ2M4TFMFZW63VQOJSXM2LFO5PXEZLROVSXG5DFMSSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/AC2DPOOSZIMUQBJHZ4EGTJT5ITPUJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRQHA3TINZVGAZ2M4TFMFZW63VQOJSXM2LFO5PXEZLROVSXG5DFMSSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because your review was requested.Message ID: ***@***.***>

[mattlevine22]

Sounds good to me. something like:

with ComputeDiagnostics(observation_scoring_config=..., filter_scoring_config=...)

given that your F/U/M thing is different...and generally, we will have different diagnostics to run (including one just on parameter posterior sampling stuff for example)

[DanWaxman]

That sounds good to me! Though it’s a bit less clear to me how we should make distinctions between metrics and diagnostics, for example. Maybe metrics should be separate, and configs for metrics can take in a lead time (which, for this PR, we can take to be lag 0 or lag 1, deferring a more complex version which pairs simulation to later)?

…

On Thu, Aug 6, 2026 at 4:21 PM Matt Levine ***@***.***> wrote: *mattlevine22* left a comment (BasisResearch/dynestyx#296) <#296 (comment)> Sounds good to me. something like: with ComputeDiagnostics(observation_scoring_config=..., filter_scoring_config=...) ? — Reply to this email directly, view it on GitHub <#296?email_source=notifications&email_token=AC2DPONPUDQH76QZIIOH5BT5ITSC7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRQHA4DMOJRGM32M4TFMFZW63VQOJSXM2LFO5PXEZLROVSXG5DFMSSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5208869137>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2DPOJB5XJD2YTEJPNGXBL5ITSC7AVCNFSNUABGKJSXA33TNF2G64TZHMYTAOJUGM4TIOBXGE5US43TOVSTWNJQG4ZTSOBYGEYTJILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AC2DPOMKWRJBXTXDYFJP6QL5ITSC7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRQHA4DMOJRGM32M4TFMFZW63VQOJSXM2LFO5PXEZLROVSXG5DFMSSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/AC2DPOPJEA3MDG5TORCN32T5ITSC7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRQHA4DMOJRGM32M4TFMFZW63VQOJSXM2LFO5PXEZLROVSXG5DFMSSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because your review was requested.Message ID: ***@***.***>

[mattlevine22]

oh hmmm interesting, it is true that this is doing eval of the 1-step-ahead forecast.

I'm not really clear on the difference between "metrics" and "diagnostics", and I'm not convinced that it will be clear to a user either, so I think they probably ought to be packaged together (or we attach diagnostics to existing "runs", like Filter, automatically).

• I get that a diagnostic is more like ESS in sampling land and a metric would be more like a suite of posterior-predictive checks (e.g., CRPS of forecast at lead time tau)

• reminds me that eventually we should also start thinking about having held-out data too (automatic splits and metrics/diagnostics computed therein as well)

[DanWaxman]

I'm not really clear on the difference between "metrics" and "diagnostics", and I'm not convinced that it will be clear to a user either, so I think they probably ought to be packaged together (or we attach diagnostics to existing "runs", like Filter, automatically).

I think, generally speaking, diagnostics are indications that computation/approximation were okay. E.g., low ESS or spread-error ratio would indicate a poor filter performance, but say very little about actual performance (other than you might expect it to be bad). Metrics are a more direct evaluation of performance, but are less interpretable usually.

I would prefer to keep them separate, since they're somewhat semantically different. This is also in line with something like arviz, which keeps things like MCMC diagnostics separate from things like loo metrics. I would also prefer to not attach them to the filter, since you probably wouldn't want to compute these during expensive inference like MCMC, and computation of many are embarrassingly parallel. I view this as a post-processing step that the filter is not particularly aware of.