bayesmith.evaluation.checks
Source-derived signatures and contracts. See the reference index for coverage and conventions.
Module contract
Prior and posterior predictive checks: does the model generate data like this? One question, asked twice from different parameter sources, and answered as an :class:`~bayesmith.artifacts.reports.EvaluationReport` on §0 ruling 7's two axes. A *posterior* predictive check reads a :class:`~bayesmith.artifacts.results.PredictiveResult` -- replicated datasets drawn at the posterior's own draws -- and asks whether the observed data look like one of them. A *prior* predictive check reads a :class:`~bayesmith.artifacts.results.SimulationResult` whose parameter source is the PRIOR and asks the same question of the prior predictive distribution, which is a question about the priors rather than about the fit. **A PASS here does not mean the model is correct, and this module is required to say so where it cannot be skipped over.** Measured, on this checkout, by ``docs/probes/probe_28_model_checking_seams.py`` §1 and re-measured through the typed path while writing this file: a straight line fitted to data with a real quadratic term of curvature 0.6 scores ``p = 0.0000`` on ``sd`` and on ``residual_sd`` -- caught, loudly. The SAME model with curvature 0.15 scores ``0.5495`` and ``0.3410``: a misspecification of exactly the kind this check exists to find, sitting comfortably inside the band. So the honest reading of a pass is "these statistics of these replicated datasets do not separate the model from the data", and nothing wider. It is written into every report's ``meta.summary`` as well as here, because a summary travels with the artifact and a module docstring does not. **The discrepancy contract (§0.3).** A discrepancy is a callable ``(y, loc) -> one scalar per draw``. Both arguments arrive already flattened to ``(draws, units)``: ``y`` is the replicated (or the observed, broadcast) data and ``loc`` is the observation's mean at the same draw, recomputed from the latent draws through :func:`~bayesmith.exact.gaussian.observation_parts` -- the same loc/scale walk R2's predictive seam uses, never a second forward model. Five defaults are provided; four read only ``y`` and the fifth reads ``y - loc``. A user callable is recorded by its IMPORTABLE IDENTITY and never as an object: see :func:`discrepancy_identity`. **Two registered thresholds and no third.** :data:`ALPHA <bayesmith.evaluation.ALPHA>` (D104) is the declared two-sided false-positive rate; :data:`DRAW_FLOOR` (D105) is derived from it. Everything else here -- the band, the tail mass, the weighting -- is arithmetic on those two. **The report-level rate is not α, and the reason is arithmetic.** §0.4 fixes the band at ``[α/2, 1 - α/2]`` PER discrepancy and reserves Bonferroni for held-out points and SBC coordinates, so a report over ``K`` cells fails a correct model with probability up to ``1 - (1 - α)**K`` -- 0.23 at the default five. That is the frozen ruling's consequence, not a defect to be corrected here by inventing a factor §0.4 did not grant; what it obliges is that the number be COUNTABLE, which is why every cell gets its own finding and why the close-out multiplies cells by α rather than guessing. This package's own acceptance tests pin measured p-values at fixed seeds, so none of them is a random trial in the first place. Layering: ``artifacts``, ``exact.gaussian``, ``graph`` and this package's own ``ALPHA``. Deliberately NOT ``dispatch``: ``tests/test_layering.py``'s ``test_graph_is_the_foundation_and_dispatch_is_the_top`` asserts that nothing depends on ``dispatch`` at module scope, and this module needs nothing from it that ``exact.gaussian`` does not already provide.
DRAW_FLOOR DEFAULT_DISCREPANCIES PRODUCER mean sd smallest largest residual_sd discrepancy_identity tail_mass_within_rate draws_resolve_the_band posterior_predictive_check prior_predictive_check
DRAW_FLOOR
Defined in bayesmith.evaluation.checks, line 138.
DRAW_FLOOR: int = math.ceil(1.0 / (ALPHA / 2.0))DEFAULT_DISCREPANCIES
Defined in bayesmith.evaluation.checks, line 218.
DEFAULT_DISCREPANCIES: tuple[Callable[..., Any], ...] = (mean, sd, smallest, largest, residual_sd)PRODUCER
Defined in bayesmith.dispatch.task, line 183.
PRODUCER = ProducerRef(package='bayesmith', version=__version__)mean
Defined in bayesmith.evaluation.checks, line 174.
mean(y: np.ndarray, loc: np.ndarray) -> np.ndarrayThe mean of the observations, per draw. Reads ``y`` only.
sd
Defined in bayesmith.evaluation.checks, line 180.
sd(y: np.ndarray, loc: np.ndarray) -> np.ndarrayThe spread of the observations, per draw. Reads ``y`` only.
smallest
Defined in bayesmith.evaluation.checks, line 186.
smallest(y: np.ndarray, loc: np.ndarray) -> np.ndarrayThe minimum observation, per draw. §0.3's ``min``. Reads ``y`` only. Spelled ``smallest`` rather than ``min`` because a module-level ``min`` would shadow the builtin this module calls a few lines away, and the recorded identity of a discrepancy is its module and qualname -- so the name is not a private detail that can be renamed later without notice.
largest
Defined in bayesmith.evaluation.checks, line 198.
largest(y: np.ndarray, loc: np.ndarray) -> np.ndarrayThe maximum observation, per draw. §0.3's ``max``. Reads ``y`` only.
residual_sd
Defined in bayesmith.evaluation.checks, line 204.
residual_sd(y: np.ndarray, loc: np.ndarray) -> np.ndarrayThe spread of ``y - loc``, per draw. The one default that reads ``loc``. The sharpest of the five on this package's own fixtures, and the reason the default set is not "four functions of the data": a quadratic term the linear model cannot express leaves the data's residuals wider than any replicated dataset's, which is what scores 0.0000 on ``curved_line(0.6)``.
discrepancy_identity
Defined in bayesmith.evaluation.checks, line 227.
discrepancy_identity(discrepancy: object) -> str``module.qualname``, and only when importing it back returns THIS object.
§0.3 says a user discrepancy is recorded by its importable identity, on
:func:`bayesmith.dispatch.amortized._embed_identity`'s rule -- and adds
that a lambda or a REPL function is refused. Those two clauses do not
agree as written, which was measured rather than assumed:
``_embed_identity`` reads ``__module__`` and ``__qualname__`` and refuses
only when one of them is empty, and a lambda has ``__qualname__ ==
'<lambda>'``, which is not empty. Handed one, it returns
``'__main__.<lambda>'`` -- an address that will never resolve again.
So the rule here is the same identity with the missing half supplied: the
identity must RESOLVE, by importing the module and walking the qualname,
and the object it resolves to must be the one that was passed. That
refuses a lambda (``getattr(module, '<lambda>')`` does not exist), a
closure (``'outer.<locals>.inner'``), a REPL function (``__main__`` has no
file to re-import from in another process), and the subtler case neither
of the first two covers: a module-level name that has since been rebound
to something else, where the address survives and its meaning does not.
Raises:
ValueError: when there is no identity, or it does not resolve back.tail_mass_within_rate
Defined in bayesmith.evaluation.checks, line 152.
tail_mass_within_rate(tail_mass: float) -> boolD104: ``tail_mass >= ALPHA / 2`` -- the p-value is inside the band. ``tail_mass`` is ``min(p, 1 - p)``, so this ONE comparison is the whole two-sided band ``[ALPHA/2, 1 - ALPHA/2]``: a p below the lower edge and a p above the upper edge both give a tail mass below ``ALPHA/2``. Written as one number against one threshold rather than as two comparisons because the report records what was compared, and ``1 - p >= ALPHA/2`` and ``p <= 1 - ALPHA/2`` are not the same floating-point question -- a reader recomputing the verdict from the finding would occasionally get the other answer at the last ULP. Closed at the boundary: the declared rate is the mass a correct model is ALLOWED to put in the tails, so a p-value sitting exactly on the edge has not exceeded it.
draws_resolve_the_band
Defined in bayesmith.evaluation.checks, line 141.
draws_resolve_the_band(draws: int) -> boolD105: ``draws >= DRAW_FLOOR`` -- this many draws can resolve a tail. Closed at the boundary: at exactly ``DRAW_FLOOR`` draws the resolution ``1/N`` equals the tail width ``ALPHA/2``, so a p-value can reach the tail's edge, which is the least this check needs to be able to say anything. Below it the tail is narrower than one draw.
posterior_predictive_check
Defined in bayesmith.evaluation.checks, line 611.
posterior_predictive_check(graph: Graph, predictive: PredictiveResult, *, source_posterior: PosteriorResult, discrepancies: Sequence[Callable[..., Any]]=DEFAULT_DISCREPANCIES) -> EvaluationReportDo the observed data look like the model's replicated datasets? (§0.2)
Args:
graph: the model the predictive result was produced from. It carries
the observed data and the loc/scale walk; the result carries the
draws.
predictive: the subject. Its ``replicated_draws`` are the replicated
datasets and its ``meta`` is what the report's ``subject_ref``
points at.
source_posterior: the posterior ``predictive.source_posterior_ref``
names. Required, and checked by id and revision, for the same
reason :func:`bayesmith.dispatch.task.execute_task` requires it:
there is no artifact store to load one from, and §0.3 computes the
p-value with the SOURCE posterior's weights -- a weighted
posterior scored as though it were iid is a different number that
looks exactly like this one.
discrepancies: callables on §0.3's ``(y, loc) -> per-draw scalar``
contract, defaulting to :data:`DEFAULT_DISCREPANCIES`. Each is
recorded by :func:`discrepancy_identity` and never as an object.
Returns:
An :class:`~bayesmith.artifacts.reports.EvaluationReport` of kind
``posterior_predictive_check``: APPLICABLE and PASS when every cell's
p-value is inside the declared band, APPLICABLE and FAIL when one is
outside, APPLICABLE and ABSTAIN below :data:`DRAW_FLOOR` draws, and
UNVERIFIABLE where the discrepancy has no definition on this subject.
Raises:
TypeError: for the wrong kind of argument, for a source posterior that
is not the one the result names, or for an empty discrepancy set.
ValueError: for a discrepancy with no importable identity.prior_predictive_check
Defined in bayesmith.evaluation.checks, line 696.
prior_predictive_check(graph: Graph, simulation: SimulationResult, *, discrepancies: Sequence[Callable[..., Any]]=DEFAULT_DISCREPANCIES) -> EvaluationReportCould the PRIORS have generated data like this? (§0.2)
The same procedure as :func:`posterior_predictive_check` over a different
parameter source, and a different question: this one is answerable before
any fit, and what it catches is a prior so wide (or so narrow) that the
data it implies are nothing like the data in hand. Measured on this
checkout: ``straight_line``'s own ``w ~ N(0, 2)`` puts all five default
cells inside the band, and the same model under ``w ~ N(0, 1e6)`` saturates
``sd`` at 1.0000 and ``residual_sd`` at 0.0000 -- while ``mean``,
``smallest`` and ``largest`` stay comfortably inside it, which is the
limited-power sentence at the top of this module, measured again.
Args:
graph: the model, carrying the observed data the prior draws are
compared against.
simulation: the subject. Its ``parameter_source`` must be the PRIOR:
a simulation from a posterior or from fixed values answers a
different question, and is reported INAPPLICABLE rather than
silently scored as though it were this one.
discrepancies: as in :func:`posterior_predictive_check`.
Returns:
An :class:`~bayesmith.artifacts.reports.EvaluationReport` of kind
``prior_predictive_check``. Prior draws are iid, so every draw
carries weight ``1/N``: there is no source posterior to read weights
from and none is invented.
Raises:
TypeError: for the wrong kind of argument, or an empty discrepancy set.
ValueError: for a discrepancy with no importable identity.