bayesmith.optimize
Source-derived signatures and contracts. See the reference index for coverage and conventions.
Module contract
Gradient MAP -- the exit an exact solve does not have. ``wiener_solve`` and ``iterative_gls`` answer a model that is affine in its latents; everything else has only a gradient. This module is that other exit: descend the NEGATIVE joint log-density and report where you got to. **The objective is the FULL density, and that is a decision rather than an implementation detail** (D7 in the migration ledger). For a prediction-dependent sigma the joint carries a ``sum log sigma`` term that a GLS-flavoured potential drops, and dropping it moves the optimum -- so a point estimate and a draw from the same declaration would target different distributions. :func:`fit` reads :func:`~bayesmith.graph.evaluate.log_joint`, which is also what ``to_numpyro`` samples, so there is one density and not two. Three things are public, and the split is the one the callers need: * :func:`minimize` -- the optimiser on any scalar objective. A calibrator scoring a PREDICTION against DATA is not maximising a joint, and forcing it through a graph would mean inventing a model for a least-squares fit. * :func:`fit` -- the graph entry. Joint MAP over every latent, or block coordinate over ``names=`` with the rest held, which is what a gradient block inside a Gibbs sweep is. * :func:`check_loss_sense` -- the direction guard. A log-density has an error's signature and the opposite optimum, so a minimiser handed one descends a function unbounded below while the loss history looks like textbook convergence. **There is a convergence verdict, and only when it is asked for.** Until 0.10.0 there was none: ``steps`` steps were taken and the objective reached was reported, and a caller who needed a verdict compared it against something of their own. That is still what a call without ``certify=`` does, and it is still what every existing caller gets. Given ``certify=<limit>``, :func:`minimize` and :func:`fit` return a :class:`Fit` whose :attr:`~Fit.converged` is a PROOF: the Newton decrement ``sqrt(g^T H^-1 g)`` at the returned point is bounded above by ``limit``, in units of the curvature's own standard deviation -- for a negative log posterior, the posterior sigma. The Hessian is never formed from the model, every product being a ``jvp`` of a ``grad``, and the inexactness of the solve behind ``H^-1 g`` is bounded from the recomputed residual, so a truncated iteration can only make the verdict REFUSE and never approve. Where the arithmetic cannot support the claim -- an unproven curvature floor, a solve that did not converge, a precision in which the decrement's digits are rounding -- the answer is a refusal with its reason (:attr:`Fit.refusal`), not a certificate. :mod:`bayesmith.optimize.certify` is where that lives and says at length what each piece rests on. The verdict bounds the Newton decrement of the local quadratic model at this point. For a quadratic objective this is a distance to its minimum; for a nonlinear objective it is a local statement, not proof of a global MAP. It says nothing about whether the objective is the right one, nor -- with ``names=`` -- about anything beyond that block's conditional objective.
MAXIMIZE
Defined in bayesmith.optimize, line 108.
MAXIMIZE: str = 'maximize'MINIMIZE
Defined in bayesmith.optimize, line 105.
MINIMIZE: str = 'minimize'Decrement
Defined in bayesmith.optimize.certify, line 163.
class Decrement
estimate: float
distance: float
lambda2: float
residual: float
reach: float
kappa: float
products: int
dense: bool
floor_source: str
status: intOne Newton decrement: what it measured, and how well.
Attributes:
estimate: ``sqrt(g^T x)`` for the computed ``x``, in units of the
curvature's standard deviation -- the distance to the minimum of
the objective's quadratic model, as the solve left it, and a
lower bound on it.
distance: the upper bound on that distance, ``inf`` when the residual
or the curvature cannot bound it. This is what a verdict reads.
lambda2, residual, reach, kappa: ``g^T x``; the true residual's size
relative to ``g``; that residual divided by the square root of
the curvature floor, which is the ``R`` of the bound; and the
condition number the rounding allowance was computed from.
products: Hessian-vector products spent, the residual's own and any
spectral probe's included.
dense: whether the Hessian was formed and solved directly.
floor_source: ``"dense"``, ``"supplied"``, ``"probe"`` or ``"none"``
-- where the curvature the bound divides by came from, and
whether it is a proof (see the module docstring).
status: one of :data:`CONVERGED`, :data:`UNREACHED`,
:data:`NONCONVEX`, :data:`NONFINITE`.Decrement.proven
proven(self) -> boolWhether the floor the bound rests on is a proof and not an estimate.
Decrement.certifies
certifies(self, limit: float) -> boolWhether the point is PROVEN within ``limit`` of the minimum. Three things, and a probed floor fails the first however good the other two look: the floor is a proof, the solve converged, and the upper bound it allows is inside ``limit``.
Fit
Defined in bayesmith.optimize, line 115.
class Fit(NamedTuple)
values: dict[str, Any]
objective: jax.Array
history: jax.Array
certificate: Decrement | None = None
limit: float | None = None
polished: int = 0Where a descent got to, and how it got there.
Attributes:
values: the fitted point. For :func:`fit` this is EVERY latent, with
the ones outside ``names`` at the values they were held at -- a
caller stepping one block of a sweep gets back a complete
environment rather than a fragment to merge itself.
objective: the objective AT :attr:`values`, evaluated after the last
step. Deliberately not ``history[-1]``, which is the value one
step earlier: reporting that one beside this point is an
off-by-one that reads as a converged fit.
history: the objective at the start of each step, shape ``(steps,)``.
So ``history[0]`` is the objective the caller started from, and
``objective < history[0]`` is what "it improved" means.
certificate: the :class:`~bayesmith.optimize.certify.Decrement`
measured at :attr:`values`, or ``None`` when no ``certify=`` was
asked for. Read :attr:`converged` rather than this, unless the
question is WHY.
limit: the distance, in units of the curvature's own standard
deviation, that the caller asked to be proven. ``None`` with no
certificate.
polished: Newton steps kept after the descent, when ``certify=``
turned the polish on. ``0`` means the point came back as the
descent left it.Fit.converged
converged(self) -> boolWhether this point is PROVEN within :attr:`limit` of the minimum. Three conditions, all required: the curvature floor the bound divides by is a proof rather than an estimate, the decrement's own solve converged, and the upper bound it allows is inside the limit. See :meth:`~bayesmith.optimize.certify.Decrement.certifies`. **``False`` is not "it was measured and it failed".** It is also what a fit that was never asked for a certificate reports, which is most of them. The two are worth telling apart and :attr:`refusal` is what tells them apart, in a sentence. This attribute is deliberately not tri-state: a caller writing ``if fit.converged:`` gets the safe reading of every case, and one who needs more asks for more.
Fit.refusal
refusal(self) -> str | NoneWhy :attr:`converged` is ``False``, or ``None`` when it is ``True``. Reports the FIRST unmet condition in the order :meth:`~bayesmith.optimize.certify.Decrement.certifies` reads them, so the sentence names the thing to fix rather than the last thing checked. **The verdict is read from :attr:`converged`, not re-derived here.** Two chains of conditions over one object can disagree, and this pair did: written as independent branches, the last one returned "outside the limit" for a point whose distance was 0.0 and whose :attr:`converged` was ``True``. Delegating makes that unrepresentable rather than merely tested for.
check_loss_sense
Defined in bayesmith.optimize, line 253.
check_loss_sense(loss_fn: Callable[[jax.Array, jax.Array], jax.Array], predicted: jax.Array, observed: jax.Array) -> NoneRefuse a scoring function whose optimum lies the way a minimiser walks.
Measured on a one-parameter gain fit with truth ``g = 1.0``, before this
guard existed::
mean_squared_error -> g = +0.9999 loss 2499 -> 0.002617
GaussianLikelihood(0.05) -> g = -30.7349 loss -3.2e7 -> -1.3e11
Both runs report a monotonically improving loss, which is the only
evidence a caller has.
The measured half evaluates the score where its direction is unambiguous:
at the PERFECT prediction, ``loss_fn(observed, observed)``. An error
attains its minimum there and a log-density its maximum, so a perfect
prediction scoring ABOVE the current one means the function increases
toward the truth. That holds for any callable, declaration or not.
Args:
loss_fn: ``f(predicted, observed) -> scalar``.
predicted: the prediction at the starting point.
observed: the data.
Raises:
StructureError: if the sense is wrong by either test, or if either
score is not finite. The last is not scope creep: NaN compares
False against everything, so treating a non-finite score as
"cannot tell, proceed" would wave through exactly the case this
guard exists for whenever it arrives with a NaN attached.fit
Defined in bayesmith.optimize, line 610.
fit(graph: Graph, at: Mapping[str, Any] | None=None, *, names: tuple[str, ...] | None=None, method: str='adam', steps: int=200, learning_rate: float=0.01, step_sizes: Mapping[str, float] | None=None, beta1: float=0.9, beta2: float=0.999, eps: float=1e-08, certify: float | None=None, floor: float | None=None, polish: bool=True) -> FitGradient MAP: maximise the graph's joint log-density over its latents.
Args:
graph: the model. Its ``log_joint`` is the objective, negated -- the
FULL density, ``sum log sigma`` and any declared ``joint_prior``
included, which is the same density ``to_numpyro`` samples (D7).
at: where to start, ``{latent: value}``. Any latent left out starts at
its prior centre, which is where
:func:`~bayesmith.dispatch.classify.block_at` anchors a block, so
the two entries agree about "before seeing anything". Defaults to
every latent there.
names: the latents to MOVE; the rest are held at their value in
``at``. ``None`` moves all of them (joint MAP). A subset is block
coordinate, which is what a gradient block inside a sweep is.
method, steps, learning_rate, step_sizes, beta1, beta2, eps: see
:func:`minimize`. ``step_sizes`` is per latent, in the latent's own
units -- a caller who knows the declared scales should use it,
because one global rate cannot serve two latents whose units are
orders of magnitude apart.
Returns:
A :class:`Fit` whose ``values`` covers EVERY latent, held ones
included.
Raises:
StructureError: for a graph with no latents, a ``names`` entry the
graph does not declare, an empty ``names``, or anything
:func:`minimize` refuses.minimize
Defined in bayesmith.optimize, line 452.
minimize(objective: Callable[[Any], jax.Array], at: Any, *, method: str='adam', steps: int=200, learning_rate: float=0.01, step_sizes: Mapping[str, float] | None=None, beta1: float=0.9, beta2: float=0.999, eps: float=1e-08, certify: float | None=None, floor: float | None=None, polish: bool=True) -> FitDescend ``objective`` from ``at`` for ``steps`` steps.
Args:
objective: ``f(values) -> scalar``, MINIMISED. Handing it something
that should be maximised is the failure :func:`check_loss_sense`
exists for; this function cannot detect it, because a bare
objective has no perfect-prediction point to evaluate at.
at: the starting point -- any pytree, or a ``{name: value}`` mapping
if ``step_sizes`` is used.
method: ``"adam"`` or ``"gradient"``.
steps: how many to take. There is no early stop, so this is also
exactly how many are taken.
learning_rate: the step size for every leaf without one of its own.
step_sizes: per-name step sizes, in each parameter's own units.
beta1, beta2, eps: Adam's, ignored by ``"gradient"``.
certify: a distance to PROVE the returned point is within, in units of
the curvature's own standard deviation -- for a negative log
posterior, the posterior sigma. Given one, the fit carries a
:class:`~bayesmith.optimize.certify.Decrement` and
:attr:`Fit.converged` answers. Left out, nothing is measured and
:attr:`Fit.converged` is ``False`` for that reason; see
:attr:`Fit.refusal`.
floor: a verified lower bound on the smallest eigenvalue of the
objective's Hessian. Only consulted above
:data:`~bayesmith.optimize.certify.DENSE_MAX` real parameters,
where the Hessian is not formed; below it the floor is computed
and this is not needed. It is a CLAIM the caller must be able to
prove -- a model affine in every parameter jointly, with proper
Gaussian priors and a noise scale that does not depend on the
prediction, is bounded below by its prior precision. An unsound
floor makes an unsound certificate, which is the one way to get a
wrong answer here. Refused without ``certify=``.
polish: whether to take Newton steps after the descent before the
certificate is measured. Only consulted with ``certify=``, and on
by default there because without it almost nothing certifies: a
first-order step of ``rate * sign(gradient)`` does not shrink with
the gradient, so Adam settles a fraction of a step size from the
optimum -- measured upstream at 2881 curvature standard deviations
on a two-parameter power law, at every step count. Each Newton
step is kept only if it lowers the objective, so this cannot make
the answer worse. Ignored without ``certify=``, on the same rule
as ``beta1``/``beta2`` under ``method="gradient"``: refuse where
it changes the answer, honour "ignored" where the contract says
ignored.
Returns:
A :class:`Fit`.
Raises:
StructureError: for an unknown method, a non-positive step count or
rate, a NaN rate, a complex starting value, or a ``step_sizes``
naming something not being moved.
Exception: from ``eqx.error_if`` if the objective at the point
reached is not finite -- a diverged descent, which otherwise
comes back looking like any other answer.
Note:
**The two methods do not accept the same rates**, and that is
arithmetic rather than taste: plain gradient descent diverges for any
rate above ``2/L``, where ``L`` is the objective's curvature, and
Adam's step is bounded by its rate regardless. Measured on a
one-latent Gaussian with ``L = 231`` (limit 0.00865): ``"gradient"``
converges at 0.006 and returns NaN at 0.02, while ``"adam"``
converges at both. Switching ``method=`` without revisiting
``learning_rate=`` is therefore a real hazard, which is why the
non-finite result is refused rather than returned.sense_of
Defined in bayesmith.optimize, line 242.
sense_of(scoring: Any) -> str:data:`MAXIMIZE` if ``scoring`` declares it, :data:`MINIMIZE` otherwise. A declaration is cheap and exact and gives the best message, but on its own it is a whitelist -- and a whitelist is wrong about precisely the code it has not met. :func:`check_loss_sense` therefore uses this as its first half and measures the second.