bayesmith.evaluation.sbc
Source-derived signatures and contracts. See the reference index for coverage and conventions.
Module contract
Simulation-based calibration: does the route return the posterior it claims?
The one check in this layer that does not read a finished result and stop.
SBC generates its own subject: draw ``theta_true`` from the prior, draw ``y``
from the model at that ``theta_true``, infer a posterior from ``y`` alone, and
ask where ``theta_true`` fell inside it. Over many replicates those positions
must be uniform, because a correct posterior places the truth uniformly by
construction. A route that is systematically too wide puts the truth in the
middle too often; one that is too narrow pushes it into the tails.
**What a PASS here does NOT say: that the route read the data.** A "posterior"
that ignores ``y`` and hands back prior draws is uniform in rank BY
CONSTRUCTION -- the position of a prior draw among prior draws is uniform -- so
it scores APPLICABLE x PASS from this harness. Measured on this checkout with
a sampler that discards its ``datum`` and returns ``2.0 * normal(key, (n,))``,
N = 100 replicates on the straight-line fixture: KS p = 0.9532, 0.7265, 0.6004
and 0.1842 at seeds 0, 1, 3 and 4 -- PASS at every one -- with seed 2's 0.0474
the false positive ``ALPHA = 0.05`` declares in advance. SBC asks whether a
route's stated uncertainty is consistent with its stated prior; a route that is
trivially self-consistent answers yes. Something that reads the observation --
a posterior predictive check, a held-out score -- has to be reported alongside
it before anyone says a route works. This is the discipline the R3 plan's
§0.3 already imposes on PPC ("do not tell a PASS as 'the model is correct'"),
and it is owed here for the same reason. The cell that pins it is
``test_a_posterior_that_ignores_the_data_is_still_calibrated``.
A second limit of the same shape, and a property of the FIXTURE rather than of
this harness: where the likelihood dominates the prior, a posterior computed
under the WRONG prior still passes. Measured on the same fixture at N = 100, a
conjugate sampler told the prior is ``N(0, 0.5)`` while the model declares
``N(0, 2)`` gives p = 0.1166 / 0.4411 / 0.4049 at seeds 0 / 1 / 2 -- PASS at
all three -- and is only caught once the sampler's prior is wrong by a further
factor of five (``N(0, 0.1)``: p = 2.5e-15 / 2.6e-16 / 6.8e-24). What SBC has
power against is set by the fixture as much as by the replicate count, and
D106 is a floor measured against a 2x WIDTH error, not a promise about every
error.
**Three rulings from the R3 plan's §0.6 are implemented here literally, and
none of them is re-decided in this module.**
*The rank is continuous and weighted.* ``r = sum_i w_i * 1[theta_i <
theta_true]`` in ``[0, 1]``, with ``w_i = 1/n`` for an equally weighted sample
and the posterior's own normalised weights for an importance-weighted one.
The classical integer rank would need a resample to accept weights, and R2
§0.5 already ruled that a weighted result is not resampled on the way into a
judgement -- a resample would make the check's answer depend on a second RNG
that nothing declared.
*Uniformity is tested with :func:`scipy.stats.kstest`,* one test per scalar
latent coordinate, against a Bonferroni-corrected level ``alpha / K`` for the
``K`` coordinates. ``alpha`` is D104 and lives in this package's ``__init__``;
the Bonferroni factor is a derivation from it, not a second threshold.
*A replicate that did not finish makes the whole report ABSTAIN, with a
count.* Never a silently dropped replicate. SBC checks the algorithm AND its
implementation, so the replicates that refused, diverged or came back without
draws are part of what is being measured: an SBC that discards its failures
and reports uniformity over the survivors is checking a different, easier
question and calling the answer by this one's name. The three failure buckets
are counted separately (``refused``, ``unconverged``, ``undrawn``) and all
three appear in the report's findings.
**The replicate floor is D106 = 100**, and the arithmetic that fixes it is
:func:`replicates_meet_floor`'s only job. See its docstring for the
measurement -- including why the R3 plan's provisional 50 did not survive
being measured over ten seeds instead of one.
**Two arms, one rank definition (§0.11).** The route arm compiles and
executes a real :class:`~bayesmith.artifacts.tasks.PosteriorTask` per
replicate, so exact and sampled routes are covered by the same harness. The
sampler arm takes any ``sampler(datum, key, n) -> draws`` and is what an
amortized estimator is scored through. Both reach
:func:`sbc_report` with the same :class:`SbcRanks`, which is the point: a
comparison between a NUTS posterior and a neural one is only a comparison if
the two were judged by identical arithmetic.
Layering: this module reads ``dispatch``, ``graph`` and ``artifacts`` and is
read by none of them (§0.1). It creates no Result and modifies none: the
posteriors it ranks are produced by ``dispatch.task`` exactly as any other
caller would produce them.REPLICATE_FLOOR REPORT_KIND SbcRanks continuous_rank ranks_are_uniform replicates_meet_floor sbc_ranks sbc_report simulation_based_calibration
REPLICATE_FLOOR
Defined in bayesmith.evaluation.sbc, line 149.
REPLICATE_FLOOR = 100REPORT_KIND
Defined in bayesmith.evaluation.sbc, line 140.
REPORT_KIND = 'sbc'SbcRanks
Defined in bayesmith.evaluation.sbc, line 247.
class SbcRanks
coordinates: tuple[str, ...]
ranks: tuple[tuple[float, ...], ...]
requested: int
usable: int
refused: int
unconverged: int
undrawn: int
route: str
subject_ref: ArtifactRef
fingerprints: FingerprintBundleWhat the replicate loop measured, with no verdict attached. Separated from :func:`sbc_report` so that the two arms of §0.11 -- a compiled route and an arbitrary sampler -- reach one judgement through one object, and so that a caller who wants the ranks themselves (a histogram, a plot) does not have to reach inside a report to get them. ``ranks`` is one tuple per coordinate, each holding one rank per USABLE replicate; the failure counts say how many replicates are missing from those tuples and why.
SbcRanks.unusable
unusable(self) -> intReplicates that produced no rank, whatever the reason.
continuous_rank
Defined in bayesmith.evaluation.sbc, line 231.
continuous_rank(draws: np.ndarray, weights: np.ndarray, truth: float) -> float§0.6's rank: ``sum_i w_i * 1[theta_i < truth]``, in ``[0, 1]``. ``draws`` is one scalar coordinate's draws and ``weights`` the posterior's normalised weights over the same draw axis. Strict ``<`` matches the classical rank statistic's convention; for a continuous posterior the tie set has probability zero, and for a discrete one the choice would have to be made by whoever owns that discreteness rather than here.
ranks_are_uniform
Defined in bayesmith.evaluation.sbc, line 209.
ranks_are_uniform(p_value: float, level: float) -> bool``p_value >= level`` -- this coordinate's ranks are uniform enough. The comparison that turns the KS test into a verdict, given its own name so that it can be exercised and mutated directly rather than only through a hundred-replicate calibration run. ``level`` is NOT a threshold of this module's: it is ``ALPHA / K``, the Bonferroni correction §0.4 derives from D104 for the ``K`` scalar latent coordinates tested together. D104 is declared once, in this package's ``__init__``; the factor K is arithmetic. Two checks in one layer each choosing their own false-positive rate is what having one ALPHA prevents, and it is why this function takes the level rather than reaching for it. ``>=`` rather than ``>``: alpha is the size of the test -- the probability of rejecting a CORRECT posterior -- so the rejection region is ``p < alpha``, and a p-value landing exactly on the level is inside the region the test was declared to accept.
replicates_meet_floor
Defined in bayesmith.evaluation.sbc, line 173.
replicates_meet_floor(usable: int, floor: int=REPLICATE_FLOOR) -> boolD106: ``usable >= floor`` -- enough replicates to detect a 2x width error.
Closed on purpose, and at 100 rather than at the 50 the R3 plan proposed.
The plan's 50 came from probe_28 §4, which swept ONE seed per distortion.
Task 6.1 required the sweep be repeated across ten seeds
(``jax.random.key(23 + k)``, k in 0..9) and the WORST -- least detectable --
p-value be the number the floor answers to. Measured by
``docs/probes/probe_30_sbc_replicate_floor.py`` on this checkout, over the
straight-line fixture, KS p for a 2x-too-wide and a 2x-too-narrow
posterior:
====== ============== ================ =========================
N worst 2x wide worst 2x narrow verdict against alpha / 3
====== ============== ================ =========================
20 0.4813 0.7045 undetectable
50 0.1384 0.1384 crosses, and crosses alpha too
100 0.0054 0.0054 3x inside the margin
====== ============== ================ =========================
At N = 50 the worst seed puts a doubled posterior width at p = 0.1384 --
not merely past the plan's alpha/3 = 0.0167 tripwire but past alpha = 0.05
itself, so one seed in ten would have MISSED a 2x error outright. The
plan's own escape clause ("if the worst p at N = 50 crosses alpha/3, raise
the floor to 100 and record that you did") is therefore taken here, and
this docstring is the record.
``>=`` rather than ``>``: the floor is the smallest count that was measured
to work, so a run that reaches it exactly has what the measurement asked
for. An open boundary would refuse the very count the number was chosen
to name.sbc_ranks
Defined in bayesmith.evaluation.sbc, line 411.
sbc_ranks(graph: Graph, *, key: jax.Array, replicates: int, model_ref: ModelRef, build: GraphBuilder | None=None, budget: ComputeBudget | None=None, nuts_on_collapse: bool=False, sampler: Sampler | None=None, sampler_draws: int=0, subject_ref: ArtifactRef | None=None) -> SbcRanks | RefusalRun the replicate loop and return its ranks, unjudged.
Args:
graph: the model, used for the prior simulation. Its own observed data
is not read -- ``SimulationTask(PRIOR)`` draws the observations
from their own distributions.
key: PRNG key. Split once into a simulation key and a replicate key,
then ``fold_in`` per replicate, so replicate i is reproducible on
its own and adding replicates does not disturb the earlier ones.
replicates: how many ``(theta_true, y)`` pairs to draw. Never
defaulted: a replicate count is a budget, and D106 is a floor on
it rather than a value for it.
model_ref: what the model callable is, for every compiled task.
build: ``build(data) -> Graph``, the route arm. Required unless
``sampler`` is given.
budget: the per-replicate posterior budget for the route arm.
nuts_on_collapse: passed to each replicate's ``PosteriorTask``. The
default is False so that a route that collapses is COUNTED as an
unusable replicate rather than silently answered by a different
route -- SBC is a statement about one route.
sampler: the §0.11 arm. ``sampler(datum, key, n) -> {name: draws}``.
sampler_draws: ``n`` for the sampler arm.
subject_ref: required with ``sampler``, which produces no Result to
point at; ignored by the route arm, which uses the first usable
replicate's posterior (§0.2's "routing representative").
Returns:
:class:`SbcRanks`, or the ``Refusal`` that stopped the prior
simulation. A REPLICATE's refusal is never returned: §0.6 says a
refused replicate is COUNTED and the report abstains, so propagating
one would replace a verdict about a calibration run with a verdict
about its first bad draw.sbc_report
Defined in bayesmith.evaluation.sbc, line 636.
sbc_report(ranks: SbcRanks, *, alpha: float=ALPHA, floor: int=REPLICATE_FLOOR) -> EvaluationReportJudge one :class:`SbcRanks`, on §0.2's two axes. * Any replicate that produced no rank -> APPLICABLE x ABSTAIN, finding ``replicates_not_completed``, with all three counts. Checked BEFORE the floor, because "45 of 100 replicates diverged" and "you only asked for 45" are different facts and the first is the one worth reporting. * Fewer usable replicates than D106 -> APPLICABLE x ABSTAIN, finding ``replicates_below_floor``. * Otherwise every coordinate's KS p against ``alpha / K`` decides: all at or above -> PASS, any below -> FAIL. ``alpha / K`` is the Bonferroni correction §0.4 derives for K simultaneous coordinates; it is a derivation from D104 and carries no threshold of its own. The per-coordinate findings record the level they were compared against, so the verdict can be recomputed from the report -- which is §8's gate G8, and the reason the level is in ``expected`` rather than only in the message.
simulation_based_calibration
Defined in bayesmith.evaluation.sbc, line 734.
simulation_based_calibration(graph: Graph, *, key: jax.Array, replicates: int, model_ref: ModelRef, build: GraphBuilder | None=None, budget: ComputeBudget | None=None, nuts_on_collapse: bool=False, sampler: Sampler | None=None, sampler_draws: int=0, subject_ref: ArtifactRef | None=None, alpha: float=ALPHA, floor: int=REPLICATE_FLOOR) -> EvaluationReport | RefusalRun the replicates and judge them: :func:`sbc_ranks` then :func:`sbc_report`. The convenience the gate runner calls. It decides nothing the two halves do not: everything about which arm runs is in the first, everything about the verdict is in the second, and this function only refuses to judge what could not be measured.