bayesmith.evaluation.heldout
Source-derived signatures and contracts. See the reference index for coverage and conventions.
Module contract
Held-out prediction: score the observations the graph declared unseen. R3 §0.8's ruling in one function. A held-out point is not a new task parameter and not a second list of indices kept beside the data -- it is a position where an observed node's ``observed_mask`` is False. The mask is already the graph's own statement of what was conditioned on (``log_joint`` and :func:`~bayesmith.dispatch.predictive.pointwise_log_likelihood` both read it, and probe_28 §7 measured the masked positions' pointwise log-likelihood as exactly 0), so reading it here means there is ONE answer to "which points did the posterior see?" rather than two that agree until someone edits one. **Conditioning and prediction cover different sets, on purpose.** Only the True positions condition. Prediction reaches every position, because the loc/scale :func:`~bayesmith.exact.gaussian.observation_parts` reads exist at a masked position exactly as they do at a conditioned one -- nothing about the forward model knows a point was withheld. That asymmetry is the whole content of a held-out check: the predictive at a point the posterior never saw is an honest out-of-sample prediction, and comparing it to the value sitting there is the only comparison in this package that is not, in part, a comparison with itself. **Two numbers per held-out point, and they answer different questions.** * The PIT, ``F_pred(y_j) = Σ_i w_i Φ((y_j - loc_ij) / scale_ij)`` -- where the observed value falls in its own predictive distribution. Under a calibrated model it is Uniform(0, 1), which is what makes a TAIL a testable event. * ``elpd_heldout = Σ_j log Σ_i w_i p(y_j | θ_i)`` -- the log predictive density actually achieved. It is REPORTED, never thresholded: there is no scale on which "-1.02 is good and -24.05 is bad" without a second model to compare against, and model comparison is R7's (section "R3 explicit non-goals"). The verdict comes from the PIT alone; the elpd is what a later comparison will read. **The band is derived, not chosen.** §0.4 declares ONE false-positive rate for the whole layer, :data:`bayesmith.evaluation.ALPHA` (D104), and everything here is arithmetic on it: a two-sided test of one point's PIT rejects outside ``[α/2, 1 - α/2]``; m points tested together with a family-wise error of at most α is Bonferroni's ``α/m`` per point, hence ``α/(2m)`` per TAIL. No number is introduced here -- ``_tail`` is the derivation written down once so that a reader can check it and a mutation can break it. **Weights come from the source posterior, because that is where they live.** A :class:`~bayesmith.artifacts.results.PredictiveResult` carries draws and no weights; an importance-weighted posterior (the ``snis`` route) whose draws were averaged uniformly would produce a confident, wrong PIT and nothing in the report would look unusual. So the source posterior is a required argument, and it is checked against the reference the predictive result already carries rather than trusted.
REPORT_KIND
Defined in bayesmith.evaluation.heldout, line 81.
REPORT_KIND = 'held_out_prediction'held_out_report
Defined in bayesmith.evaluation.heldout, line 217.
held_out_report(graph: Graph, predictive: PredictiveResult, *, source_posterior: PosteriorResult) -> EvaluationReportScore the positions ``graph`` masked out, against ``predictive``'s draws.
Args:
graph: the model the predictive result was produced against. Its
observed nodes' ``observed_mask`` is the definition of held out.
predictive: the subject. Its carried ``latent_draws`` are replayed
through the forward model; nothing is re-sampled.
source_posterior: the posterior ``predictive.source_posterior_ref``
names. Required for its WEIGHTS, which a predictive result does
not carry -- see the module docstring.
Returns:
An :class:`~bayesmith.artifacts.reports.EvaluationReport` of kind
``held_out_prediction``, on §0.2's row:
* APPLICABLE / PASS -- every held-out point's PIT is inside
``[α/2m, 1 - α/2m]``.
* APPLICABLE / FAIL -- at least one is outside it.
* INAPPLICABLE / ABSTAIN -- the graph masks nothing out, so there is
no out-of-sample point to score. Not a PASS: a check with no
subject has verified nothing, and a gate counting it as a check that
was made would be counting an empty set.
* UNVERIFIABLE / ABSTAIN -- the predictive cannot be evaluated: a
correlated or non-Gaussian observed node (§0.4's coverage domain),
a source posterior that holds no draws, or a subject whose carried
latents do not cover the graph.
Raises:
TypeError: if the arguments are of the wrong type, or if
``source_posterior`` is not the artifact and revision
``predictive.source_posterior_ref`` names. That is a caller
error, not a verdict about a model: a report saying "unverifiable"
would file the mistake as a property of the subject.