Observation scoring PR

docs/api_reference/developer/evaluation/handlers.md

@@ -0,0 +1,5 @@
+# Evaluation
+
+`Evaluation` is an outer effectful handler. It consumes the `filtered_result` forwarded by `Filter`, delegates pure computation to `dynestyx.evaluation.observation_scoring`, attaches the resulting `EvaluationResult`, and returns its deferred registration callback through the handler stack.
+
+::: dynestyx.evaluation.handlers

docs/api_reference/developer/diagnostics/plotting_utils.md → docs/api_reference/developer/evaluation/plotting_utils.md

@@ -2,7 +2,7 @@
 
 **plot_hmm_states_and_observations**
 
-::: dynestyx.diagnostics.plotting_utils.plot_hmm_states_and_observations
+::: dynestyx.evaluation.plotting_utils.plot_hmm_states_and_observations
     options:
       show_root_heading: false
       show_root_toc_entry: false
@@ -12,7 +12,7 @@
 
 **plot_continuous_states_and_partial_observations**
 
-::: dynestyx.diagnostics.plotting_utils.plot_continuous_states_and_partial_observations
+::: dynestyx.evaluation.plotting_utils.plot_continuous_states_and_partial_observations
     options:
       show_root_heading: false
       show_root_toc_entry: false

docs/api_reference/developer/evaluation/scoring.md

@@ -0,0 +1,15 @@
+# Scoring
+
+Proper scoring rules let us evaluate predictive observation distributions w.r.t. data in ways beyond marginal likelihood.
+
+`dynestyx.evaluation.scoring` defines the score objects themselves: `BaseObservationScore`, `GaussianLogProbScore`, `DawidSebastianiScore`, `ObservationWiseCRPSScore`, and `EnergyScore`. These scores currently operate on the one-step-ahead predictive observation distributions produced by the continuous-time CD-Dynamax Gaussian filters (`ContinuousTimeKFConfig`, `ContinuousTimeEKFConfig`, `ContinuousTimeUKFConfig`, and `ContinuousTimeEnKFConfig`). `ObservationScoringConfig` is documented on the companion [Scoring Configs](../inference/configs/scoring_configs.md) page.
+
+::: dynestyx.evaluation.scoring
+    options:
+      members:
+        - BaseObservationScore
+        - GaussianLogProbScore
+        - DawidSebastianiScore
+        - ObservationWiseCRPSScore
+        - EnergyScore
+

docs/api_reference/developer/inference/configs/filter_configs.md

@@ -2,6 +2,8 @@
 
 The single `Filter()` handler is directed to the appropriate filtering algorithm via the provided `FilterConfig`.
 
+`include_predicted_observations` controls result-level collection of supported backend predictive-observation outputs. The `record_predicted_observations_mean`, `record_predicted_observations_cov`, and `record_predicted_observations_ensemble` fields separately control NumPyro recording. Collection and recording default to enabled; unavailable fields are omitted without making Filter fail.
+
 ::: dynestyx.inference.configs.filter
     options:
       filters: []

docs/api_reference/developer/inference/configs/scoring_configs.md

@@ -0,0 +1,10 @@
+# Scoring Configs
+
+`dynestyx.evaluation.configs` contains configuration for evaluating conditioned results. `ObservationScoringConfig` currently configures one-step-ahead predictive-observation scoring.
+
+The score rule objects themselves live on the companion [Scoring](../../evaluation/scoring.md) page.
+
+::: dynestyx.evaluation.configs
+    options:
+      members:
+        - ObservationScoringConfig

docs/api_reference/developer/inference/filters.md

@@ -1,8 +1,9 @@
 # Filters
 
-One of the principal functions of a dynamical systems inference engine is *filtering*, i.e., computation of the distribution \(p(x_t \mid y_{1:T}, \theta)\). In the computation of a filtering distribution, we also obtain estimates of the marginal likelihood, \(p(y_{1:T} | \theta)\), used for parameter inference/system identification. To tell `dynestyx` that a dynamical system should be processed via a filtering algorithm, we use the `Filter` class.
+One of the principal functions of a dynamical systems inference engine is *filtering*, i.e., computation of the distribution \(p(x_t \mid y_{1:t}, \theta)\). In the computation of a filtering distribution, we also obtain estimates of the marginal likelihood, \(p(y_{1:T} | \theta)\), used for parameter inference/system identification. To tell `dynestyx` that a dynamical system should be processed via a filtering algorithm, we use the `Filter` class.
+
+This module translates supported backend predictive-observation outputs into the canonical payload stored on `ConditionedResult`. It does not perform observation scoring; that responsibility belongs to the outer [`Evaluation`](../evaluation/handlers.md) handler.
 
 ::: dynestyx.inference.filters
     options:
       filters: []
-

docs/api_reference/developer/result_types.md

@@ -4,4 +4,6 @@ The standalone and handler-based APIs return dataclasses.
 
 ::: dynestyx.types.ConditionedResult
 
+::: dynestyx.types.EvaluationResult
+
 ::: dynestyx.types.SimulatedResult

docs/api_reference/public/evaluation/handlers.md

@@ -0,0 +1,25 @@
+# Evaluation
+
+`Evaluation` consumes the `ConditionedResult` forwarded by an inner `Filter`, computes the configured evaluations, and attaches an `EvaluationResult` to it.
+
+```python
+with Evaluation(observation_scoring_config=scoring_config):
+    with Filter(filter_config=filter_config):
+        result = dsx.condition(
+            "f",
+            dynamics,
+            obs_times=obs_times,
+            obs_values=obs_values,
+        )
+
+scores = result.evaluation_result.observation_scores
+```
+
+With `dsx.condition`, the scores are available on the returned result without
+adding NumPyro sites. With `dsx.sample`, `Evaluation` registers the configured
+scores as deterministic sites through the result's deferred callback.
+
+::: dynestyx.evaluation.handlers
+    options:
+      members:
+        - Evaluation

docs/api_reference/public/diagnostics/plotting_utils.md → docs/api_reference/public/evaluation/plotting_utils.md

@@ -2,7 +2,7 @@
 
 **plot_hmm_states_and_observations**
 
-::: dynestyx.diagnostics.plotting_utils.plot_hmm_states_and_observations
+::: dynestyx.evaluation.plotting_utils.plot_hmm_states_and_observations
     options:
       show_root_heading: false
       show_root_toc_entry: false
@@ -11,7 +11,7 @@
 
 **plot_continuous_states_and_partial_observations**
 
-::: dynestyx.diagnostics.plotting_utils.plot_continuous_states_and_partial_observations
+::: dynestyx.evaluation.plotting_utils.plot_continuous_states_and_partial_observations
     options:
       show_root_heading: false
       show_root_toc_entry: false

docs/api_reference/public/evaluation/scoring.md

@@ -0,0 +1,15 @@
+# Scoring
+
+Proper scoring rules let us evaluate predictive observation distributions w.r.t. data in ways beyond marginal likelihood.
+
+`dynestyx.evaluation.scoring` defines the score objects themselves: `BaseObservationScore`, `GaussianLogProbScore`, `DawidSebastianiScore`, `ObservationWiseCRPSScore`, and `EnergyScore`. These scores currently operate on the one-step-ahead predictive observation distributions produced by the continuous-time CD-Dynamax Gaussian filters (`ContinuousTimeKFConfig`, `ContinuousTimeEKFConfig`, `ContinuousTimeUKFConfig`, and `ContinuousTimeEnKFConfig`). `ObservationScoringConfig` is documented on the companion [Scoring Configs](../inference/configs/scoring_configs.md) page.
+
+::: dynestyx.evaluation.scoring
+    options:
+      members:
+        - BaseObservationScore
+        - GaussianLogProbScore
+        - DawidSebastianiScore
+        - ObservationWiseCRPSScore
+        - EnergyScore
+

docs/api_reference/public/inference/configs/filter_configs.md

@@ -2,6 +2,8 @@
 
 The single `Filter()` handler is directed to the appropriate filtering algorithm via the provided `FilterConfig`. We provide a summary below, as well as an exhaustive list of classes.
 
+`include_predicted_observations` controls whether supported backend predictive-observation outputs are collected into `ConditionedResult` and defaults to `True`. The shared `record_predicted_observations_*` fields independently control whether available means, covariances, or ensembles are recorded to the NumPyro trace; they also default to `True`. Observation scoring is configured on the separate `Evaluation` handler.
+
 ## Available filter configurations
 
 | Config class               | Time domain         | When it fits best |

docs/api_reference/public/inference/configs/scoring_configs.md

@@ -0,0 +1,10 @@
+# Scoring Configs
+
+`ObservationScoringConfig` configures how an `Evaluation` handler scores the predictive-observation outputs carried by a filtered `ConditionedResult`.
+
+The score rule objects themselves live on the companion [Scoring](../../evaluation/scoring.md) page.
+
+::: dynestyx.evaluation.configs
+    options:
+      members:
+        - ObservationScoringConfig

docs/api_reference/public/inference/filters.md

@@ -1,9 +1,10 @@
 # Filters
 
-One of the principal functions of a dynamical systems inference engine is *filtering*, i.e., computation of the distribution \(p(x_t \mid y_{1:T}, \theta)\). In the computation of a filtering distribution, we also obtain estimates of the marginal likelihood, \(p(y_{1:T} | \theta)\), used for parameter inference/system identification. To tell `dynestyx` that a dynamical system should be processed via a filtering algorithm, we use the `Filter` class.
+One of the principal functions of a dynamical systems inference engine is *filtering*, i.e., computation of the distribution \(p(x_t \mid y_{1:t}, \theta)\). In the computation of a filtering distribution, we also obtain estimates of the marginal likelihood, \(p(y_{1:T} | \theta)\), used for parameter inference/system identification. To tell `dynestyx` that a dynamical system should be processed via a filtering algorithm, we use the `Filter` class.
+
+Where supported, `Filter` includes canonical one-step-ahead predictive-observation outputs in its `ConditionedResult`. `include_predicted_observations` controls that result payload, while the `record_predicted_observations_*` fields independently control NumPyro deterministic sites. Observation scoring is performed by an outer [`Evaluation`](../evaluation/handlers.md) handler rather than by `Filter` itself.
 
 ::: dynestyx.inference.filters
     options:
       members:
         - Filter
-

docs/api_reference/public/result_types.md

@@ -4,4 +4,6 @@ The standalone and handler-based APIs return dataclasses.
 
 ::: dynestyx.types.ConditionedResult
 
+::: dynestyx.types.EvaluationResult
+
 ::: dynestyx.types.SimulatedResult

docs/deep_dives/gp_drift.ipynb

@@ -97,7 +97,7 @@
     "    ScalarDiffusion,\n",
     "    SDESimulator,\n",
     ")\n",
-    "from dynestyx.diagnostics.plotting_utils import plot_drift_field\n",
+    "from dynestyx.evaluation.plotting_utils import plot_drift_field\n",
     "import matplotlib.pyplot as plt\n",
     "from matplotlib.lines import Line2D\n",
     "import numpy as np"

docs/faq.md

@@ -279,8 +279,8 @@ joint_log_prob = dsx.log_prob(
 
 `dsx.condition` returns a
 [`ConditionedResult`](api_reference/public/result_types.md#dynestyx.types.ConditionedResult)
-containing the
-marginal log likelihood and state summaries requested by the active handler.
+containing the inference times, marginal log likelihood, state summaries, and
+per-time distributions requested by the active filter or smoother handler.
 Pass a filter or smoother config when you need a specific algorithm.
 `dsx.log_prob` returns the joint density of a fixed latent path. For a native
 SDE latent-path workflow, first choose a
@@ -344,8 +344,8 @@ tutorial](tutorials/gentle_intro/03_filtering_mll_no_numpyro.ipynb), the
 example](tutorials/state_space_models/kf_tracking.ipynb), and the
 [NumPyro-free differentiable optimization
 tutorial](tutorials/gentle_intro/05_svi_no_numpyro.ipynb). The [result-type
-reference](api_reference/public/result_types.md) documents
-`SimulatedResult` and `ConditionedResult`.
+reference](api_reference/public/result_types.md) documents `SimulatedResult`
+and `ConditionedResult`.
 
 ## What about hierarchical models?
 

docs/tutorials/gentle_intro/00_index.ipynb

@@ -45,7 +45,9 @@
     "\n",
     "13. **[Part 11b: Missing observations with `LatentPathBuilder` + MCMC](../11b_missing_observations_latent_path_mcmc/)** — Joint posterior inference over parameters and latent states under explicit latent-path construction, including partial and full missingness.\n",
     "\n",
-    "14. **[Part 11c: Missing observations in HMMs](../11c_missing_observations_hmms/)** — Exact HMM missingness handling for `MultivariateNormal` and `Independent(..., 1)` observation families."
+    "14. **[Part 11c: Missing observations in HMMs](../11c_missing_observations_hmms/)** — Exact HMM missingness handling for `MultivariateNormal` and `Independent(..., 1)` observation families.\n",
+    "\n",
+    "15. **[Part 12: Observation scoring with filters](../12_observation_scoring_with_filters/)** — Use filter-predicted observation distributions to compute Gaussian log-probs, Dawid-Sebastiani scores, CRPS, and energy scores."
    ]
   },
   {

docs/tutorials/gentle_intro/07_hmm.ipynb

@@ -189,7 +189,7 @@
         }
       ],
       "source": [
-        "from dynestyx.diagnostics.plotting_utils import plot_hmm_states_and_observations\n",
+        "from dynestyx.evaluation.plotting_utils import plot_hmm_states_and_observations\n",
         "\n",
         "plot_hmm_states_and_observations(\n",
         "    times=obs_times[:100],\n",

docs/tutorials/gentle_intro/11c_missing_observations_hmms.ipynb

@@ -806,7 +806,7 @@
    "source": [
     "**Previous:** [Part 11b — Missing observations with `LatentPathBuilder` + MCMC](../11b_missing_observations_latent_path_mcmc/)\n",
     "\n",
-    "**Next:** [Part 12 — Hierarchical modeling patterns](../12_hierarchical_modeling_patterns/)\n"
+    "**Next:** [Part 12 — Observation scoring with filters](../12_observation_scoring_with_filters/)\n"
    ]
   }
  ],