bayesmithDocumentation0.10.0 source

bayesmith.evaluation.heldout

Source-derived signatures and contracts. See the reference index for coverage and conventions.

Module contract
Held-out prediction: score the observations the graph declared unseen.

R3 §0.8's ruling in one function.  A held-out point is not a new task
parameter and not a second list of indices kept beside the data -- it is a
position where an observed node's ``observed_mask`` is False.  The mask is
already the graph's own statement of what was conditioned on
(``log_joint`` and
:func:`~bayesmith.dispatch.predictive.pointwise_log_likelihood` both read it,
and probe_28 §7 measured the masked positions' pointwise log-likelihood as
exactly 0), so reading it here means there is ONE answer to "which points did
the posterior see?" rather than two that agree until someone edits one.

**Conditioning and prediction cover different sets, on purpose.**  Only the
True positions condition.  Prediction reaches every position, because the
loc/scale :func:`~bayesmith.exact.gaussian.observation_parts` reads exist at a
masked position exactly as they do at a conditioned one -- nothing about the
forward model knows a point was withheld.  That asymmetry is the whole content
of a held-out check: the predictive at a point the posterior never saw is an
honest out-of-sample prediction, and comparing it to the value sitting there
is the only comparison in this package that is not, in part, a comparison with
itself.

**Two numbers per held-out point, and they answer different questions.**

* The PIT, ``F_pred(y_j) = Σ_i w_i Φ((y_j - loc_ij) / scale_ij)`` -- where the
  observed value falls in its own predictive distribution.  Under a calibrated
  model it is Uniform(0, 1), which is what makes a TAIL a testable event.
* ``elpd_heldout = Σ_j log Σ_i w_i p(y_j | θ_i)`` -- the log predictive density
  actually achieved.  It is REPORTED, never thresholded: there is no scale on
  which "-1.02 is good and -24.05 is bad" without a second model to compare
  against, and model comparison is R7's (section "R3 explicit non-goals").  The verdict
  comes from the PIT alone; the elpd is what a later comparison will read.

**The band is derived, not chosen.**  §0.4 declares ONE false-positive rate for
the whole layer, :data:`bayesmith.evaluation.ALPHA` (D104), and everything here
is arithmetic on it: a two-sided test of one point's PIT rejects outside
``[α/2, 1 - α/2]``; m points tested together with a family-wise error of at
most α is Bonferroni's ``α/m`` per point, hence ``α/(2m)`` per TAIL.  No number
is introduced here -- ``_tail`` is the derivation written down once so that a
reader can check it and a mutation can break it.

**Weights come from the source posterior, because that is where they live.**  A
:class:`~bayesmith.artifacts.results.PredictiveResult` carries draws and no
weights; an importance-weighted posterior (the ``snis`` route) whose draws were
averaged uniformly would produce a confident, wrong PIT and nothing in the
report would look unusual.  So the source posterior is a required argument, and
it is checked against the reference the predictive result already carries
rather than trusted.

REPORT_KIND

Defined in bayesmith.evaluation.heldout, line 81.

REPORT_KIND = 'held_out_prediction'

held_out_report

Defined in bayesmith.evaluation.heldout, line 217.

held_out_report(graph: Graph, predictive: PredictiveResult, *, source_posterior: PosteriorResult) -> EvaluationReport
Score the positions ``graph`` masked out, against ``predictive``'s draws.

Args:
    graph: the model the predictive result was produced against.  Its
        observed nodes' ``observed_mask`` is the definition of held out.
    predictive: the subject.  Its carried ``latent_draws`` are replayed
        through the forward model; nothing is re-sampled.
    source_posterior: the posterior ``predictive.source_posterior_ref``
        names.  Required for its WEIGHTS, which a predictive result does
        not carry -- see the module docstring.

Returns:
    An :class:`~bayesmith.artifacts.reports.EvaluationReport` of kind
    ``held_out_prediction``, on §0.2's row:

    * APPLICABLE / PASS -- every held-out point's PIT is inside
      ``[α/2m, 1 - α/2m]``.
    * APPLICABLE / FAIL -- at least one is outside it.
    * INAPPLICABLE / ABSTAIN -- the graph masks nothing out, so there is
      no out-of-sample point to score.  Not a PASS: a check with no
      subject has verified nothing, and a gate counting it as a check that
      was made would be counting an empty set.
    * UNVERIFIABLE / ABSTAIN -- the predictive cannot be evaluated: a
      correlated or non-Gaussian observed node (§0.4's coverage domain),
      a source posterior that holds no draws, or a subject whose carried
      latents do not cover the graph.

Raises:
    TypeError: if the arguments are of the wrong type, or if
        ``source_posterior`` is not the artifact and revision
        ``predictive.source_posterior_ref`` names.  That is a caller
        error, not a verdict about a model: a report saying "unverifiable"
        would file the mistake as a property of the subject.

Type a concept or API name.