bayesmith.evaluation.gate
Source-derived signatures and contracts. See the reference index for coverage and conventions.
Module contract
``model_checking@1``: eight reports about one posterior, aggregated once.
This module is the only one in the layer that reaches no verdict of its own,
and that is its whole design (§8.2). Every question it asks already has an
owner: the predictive band is :mod:`bayesmith.evaluation.checks`'s, the
replicate floor is :mod:`bayesmith.evaluation.sbc`'s, the rank tolerance and
the shift criterion are :mod:`bayesmith.diagnose`'s, and the truth table that
turns eight reports into one status and one verdict is
:func:`bayesmith.artifacts.gates.aggregate_gate`'s. Nothing here compares a
number against a constant. If a reader finds one, it is a bug: it means a
decision that lives somewhere else has been copied, and a copy agrees with its
original right up until somebody edits one of them.
**Two checks are required and six are optional, and the split is not about
cost.** ``posterior_predictive_check`` asks whether the fitted model can
reproduce the data it was fitted to, and ``identifiability`` asks whether the
parameters were determined by that data at all; a posterior that fails either
has not been checked in any useful sense, so their absence must leave the gate
undecided. The other six each answer a question that is real but conditional
-- there may be no held-out point, no ArviZ, no calibration campaign -- and
§0.6 lets only required checks decide.
**The trap this runner exists to avoid, measured before it was written.**
:func:`~bayesmith.artifacts.gates.aggregate_gate` was probed on this checkout
with one required PASS slot beside one optional slot in six states::
optional slot ABSENT from slots status=evaluated verdict=pass n_refs=1 findings=[]
optional slot NOT_ATTEMPTED status=evaluated verdict=pass n_refs=1 findings=[]
optional slot UNVERIFIABLE x ABSTAIN status=evaluated verdict=pass n_refs=2 findings=[]
optional slot APPLICABLE x ABSTAIN status=evaluated verdict=pass n_refs=2 findings=[]
optional slot APPLICABLE x PASS status=evaluated verdict=pass n_refs=2 findings=[]
optional slot APPLICABLE x FAIL status=evaluated verdict=pass n_refs=2 findings=['optional_report_failed']
So on the OPTIONAL axis the status, the verdict and the findings are identical
whether a check was never attempted or was attempted and could not answer:
``report_refs`` is the one field that separates them. (On the REQUIRED axis
they are separate findings -- ``required_report_missing`` against
``required_report_abstained`` -- which is why the trap is an optional-slot
trap.) **Therefore this runner never drops a report.** A check that ran files
its report whatever the report says; only a check that nobody ran gets
``NOT_ATTEMPTED``. A runner that skipped abstaining optional reports because
"they change nothing" would emit a byte-identical clean PASS for a run where a
check was attempted and failed to answer.
**Two of the eight are CARRIED rather than run**, and the reason is the same
ruling that keeps numbers out of this file. ``sbc`` costs at least
:data:`~bayesmith.evaluation.sbc.REPLICATE_FLOOR` posterior fits, so running it
here would mean this module choosing a replicate count -- a number, and one
whose floor already has a home. ``chain_diagnostics`` is decided by the
execution layer while it makes the posterior -- re-deciding convergence here
would put a second owner on it (§2.4, red line 3) -- and only its REFERENCE
survives on
:attr:`~bayesmith.artifacts.results.PosteriorResult.report_refs`, so a caller
who wants that slot filled supplies the report. A caller with neither gets
two ``NOT_ATTEMPTED`` slots, which is the honest record.
**A check that raises becomes an ``ErrorRecord``, never an exception.** Not a
defensive habit -- a measured requirement. On this checkout ``arviz.loo``
raises ``ValueError: n_draws_tail must be at least 5`` on an 8-draw posterior
and ``ValueError: All tail values are the same`` on an 8-point masked line, so
the ``draws=8`` fixture §8.1 asks for cannot reach a verdict at all unless the
runner catches. ``ReportSlot`` has the field for exactly this, and §0.6
decides what an error means: blocking for a required check, ignored for an
optional one unless the requirement says otherwise. What this module must not
do is let the exception out, because then a gate that could not run a check
produces no record instead of a record saying so.
**``prerequisites_ready`` and ``inputs_current`` are READ, not decided.** The
first is
:attr:`~bayesmith.artifacts.results.PosteriorResult.predictive_ready`, which is
the execution layer's own statement that a predictive result can be made from
this posterior -- and three of the eight checks need one, so when it is False
§0.6's BLOCKED ("needs an artifact that does not exist yet") is literally the
case. The second is the posterior's lifecycle status: reports about an
invalidated subject judge something that has moved.
Layering: ``dispatch``, ``graph``, ``artifacts`` and this package's own
sibling modules (§0.1). ArviZ stays optional -- it is reached only through
:func:`bayesmith.evaluation.loo.loo_report`, which imports it inside the
function that needs it.CARRIED_KINDS CHAIN_DIAGNOSTICS MODEL_CHECKING POSTERIOR_PREDICTIVE_CHECK PRIOR_PREDICTIVE_CHECK check_posterior model_checking_slots
CARRIED_KINDS
Defined in bayesmith.evaluation.gate, line 171.
CARRIED_KINDS: tuple[str, ...] = (SBC, CHAIN_DIAGNOSTICS)CHAIN_DIAGNOSTICS
Defined in bayesmith.evaluation.gate, line 165.
CHAIN_DIAGNOSTICS = 'chain_diagnostics'MODEL_CHECKING
Defined in bayesmith.evaluation.gate, line 196.
MODEL_CHECKING = GateDefinition(name='model_checking', version=1, requirements=(ReportRequirement(name=POSTERIOR_PREDICTIVE_CHECK, required=True), ReportRequirement(name=IDENTIFIABILITY, required=True), ReportRequirement(name=PRIOR_PREDICTIVE_CHECK, required=False), ReportRequirement(name=HELD_OUT_PREDICTION, required=False), ReportRequirement(name=LOO_PSIS, required=False), ReportRequirement(name=SBC, required=False), ReportRequirement(name=PRIOR_SENSITIVITY, required=False), ReportRequirement(name=CHAIN_DIAGNOSTICS, required=False)), blocked_actions=(), remedies=(Remedy(action='raise_draws_to_the_resolution_floor', message="A predictive check below D105's draw floor abstains rather than passing, because a p-value from that many draws cannot land in the tail the declared rate reserves. Re-run the posterior with at least this many draws.", parameters=(('draws', DRAW_FLOOR),)), Remedy(action='build_the_graph_inside_enable_x64', message='identifiability and prior_sensitivity refuse a float32 ambient precision, and they refuse a graph whose constants were traced outside the block as well (§0.10). Build the graph and run the check inside jax.enable_x64(True).')))POSTERIOR_PREDICTIVE_CHECK
Defined in bayesmith.evaluation.gate, line 161.
POSTERIOR_PREDICTIVE_CHECK = 'posterior_predictive_check'PRIOR_PREDICTIVE_CHECK
Defined in bayesmith.evaluation.gate, line 162.
PRIOR_PREDICTIVE_CHECK = 'prior_predictive_check'check_posterior
Defined in bayesmith.evaluation.gate, line 581.
check_posterior(graph: Graph, posterior: PosteriorResult, *, key: jax.Array, budget: ComputeBudget, model_ref: ModelRef, carried: Sequence[EvaluationReport]=()) -> GateResultRun ``model_checking@1`` over one posterior: slots, then the aggregator.
:func:`model_checking_slots` then
:func:`~bayesmith.artifacts.gates.aggregate_gate`, and nothing between them
-- the same split as :func:`~bayesmith.evaluation.sbc.simulation_based_calibration`'s,
for the same reason: everything about what ran is in the first, everything
about what it means is in the second, and this function decides neither.
The aggregator is not wrapped, extended or second-guessed here (§8.2).
Args:
graph: as :func:`model_checking_slots` takes it.
posterior: the subject. Two of its fields reach the aggregator
directly: ``predictive_ready`` as ``prerequisites_ready``, and the
lifecycle status of its envelope as ``inputs_current``.
key: PRNG key for the two internal runs.
budget: what those runs may spend.
model_ref: what the model callable is.
carried: ``sbc`` and ``chain_diagnostics`` reports the caller holds.
Returns:
The :class:`~bayesmith.artifacts.gates.GateResult`. Its ``meta``
carries the posterior's own fingerprints and names it as a parent, so
§0.3 retires this verdict when the thing it judged changes.
The result's identity is minted, so two calls on identical inputs give
two DIFFERENT ``report_refs`` over the same decision: they are two
artifacts, and ``new_artifact_meta`` says so in as many words. What is
deterministic is everything the gate decided, which is what
``test_gate.py`` compares.model_checking_slots
Defined in bayesmith.evaluation.gate, line 437.
model_checking_slots(graph: Graph, posterior: PosteriorResult, *, key: jax.Array, budget: ComputeBudget, model_ref: ModelRef, carried: Sequence[EvaluationReport]=()) -> tuple[ReportSlot, ...]Run the checks that apply and file each outcome in its own slot.
One slot per declared requirement, in the definition's order, always: a
requirement with no slot is invisible to a reader of the result, and an
optional one with no slot is invisible to
:func:`~bayesmith.artifacts.gates.aggregate_gate` as well.
Args:
graph: the model. READ -- nothing here modifies it (§2.4). For the
two ``diagnose`` projections it must have been BUILT inside
``jax.enable_x64(True)``; outside one they file an UNVERIFIABLE
report carrying the refusal, which is a verdict the gate can
aggregate rather than an exception it cannot.
posterior: the subject. Its ``latent_names`` name the sites carried
forward, its ``predictive_ready`` says whether a predictive result
can be made at all, and its own reference is what a carried
``chain_diagnostics`` report's ``subject_ref`` is checked against.
key: PRNG key. Split once, into one stream for the predictive replay
and one for the prior simulation, so the same key gives the same
eight reports and adding a check later cannot disturb the earlier
streams.
budget: what the two internal runs may spend. Never defaulted: a draw
count is a budget, and D105 is a floor on it rather than a value
for it.
model_ref: what the model callable is, for every compiled task.
carried: reports produced elsewhere, of the kinds in
:data:`CARRIED_KINDS`. Anything else is refused by name.
Returns:
One :class:`~bayesmith.artifacts.gates.ReportSlot` per requirement of
:data:`MODEL_CHECKING`, in that definition's order.
Raises:
TypeError: for the wrong kind of argument, or a carried entry that is
not an evaluation report.
ValueError: for a carried report this gate will not accept, two
carried reports of one kind, or a carried ``chain_diagnostics``
that judges some other result.