Mutation-testing log

Every guard added in v0.2.0 was verified by breaking the implementation on purpose and confirming the corresponding test turns red, then reverting. A test that stays green under its own mutation is not a test; it is decoration.

The harness applies a textual mutation to one source file, runs the target tests, records pytest’s exit code, and restores the file from a backup.

Why this and not “the tests pass”

Two failure modes motivated it, both observed in adjacent work:

  1. A constant that coincides with an input. A test asserted a gradient equalled 3.0 while the fixture’s parameter was also 3.0. The assertion held even when the code under test returned the input unchanged. Every equality assertion in this suite is therefore checked against a constant that appears nowhere in its inputs — exp(0.5) = 1.6487…, E[x^4] = 3, the Ishigami S1 = [0.3139, 0.4424, 0] — or is written as an inequality.

  2. Self-consistency has a blind spot. Two independent sweeps once agreed on -225.65 when the truth was -364.95: both wrong, mutually consistent. So the integration suite compares against closed-form moments, never against a previous release’s output. v0.1.4’s own numbers would have been an especially bad baseline here, since two of its four integrators returned the right answer multiplied by the box volume.

Results

17 mutations, run by bash scripts/mutation_check.sh. Sixteen turn the suite red; one stays green and is documented below as defence-in-depth.

# Mutation Target Result
1 disable the holdout-degeneracy branch (if False:) TestDegenerateTargetGuard RED
2 _holdout_target_is_degenerate → always False TestDegenerateTargetGuard RED
3 remove the unit rescaling in _distance_correlation TestDistanceCorrelationRange RED
4 corrupt the dCor normalisation (denom * 0.5) TestDistanceCorrelationRange RED
5 remove the [0, 1] clip on dCor TestDistanceCorrelationRange GREEN — defensive only
6 surrogate-collapse detector → always False TestSurrogateCollapseGuard RED
7 revert DEFAULT_N_RESTARTS_OPTIMIZER to 0 TestOptimiserRestartsDefault RED
8 silence the Sobol range report TestSobolRangeInvariants RED
9 silence the ARD-bound warning TestARDBoundWarning RED
10 ARD bound counter → always 0 TestARDBoundWarning RED
11 revert the restarts default (shipped-defaults path) TestShippedDefaultsRecoverTruth RED
12 revert MC uncertainty divisor to N_samples TestMonteCarloUncertaintyCalibration RED
13 restore qmc_integral_auto self-normalise + volume TestQMCIntegralAutoAnalytic RED
14 remove the 1-D → column reshape TestIntegrandShapeHandling RED
15 remove the leading-dimension check TestIntegrandShapeHandling RED
16 ignore seed in the Sobol engine TestQMCSeedReproducibility RED
17 remove the collapsed-surrogate short-circuit before SALib TestSurrogateCollapseGuard RED

A test that could not fail on the machine that wrote it

v0.2.0 shipped a crash that its own guard was supposed to prevent, and the regression test for it would have passed locally even without the fix.

gp_surrogate_and_sobol detected a collapsed surrogate, warned, and then handed the collapsed response to SALib anyway. SALib’s first_order() returns np.array([0.0]) – a one-element array rather than a scalar – for an exactly constant response, and assigning that into its float result array is a DeprecationWarning on NumPy 1.25-2.3 but a ValueError on 2.4. The machine this was developed on had NumPy 2.3.5; CI installed 2.4.6.

So test_collapsed_fit_does_not_crash_on_modern_numpy is honest but version-conditional: on NumPy < 2.4 it passes whether or not the guard exists. It is paired with TestSALibConstantResponseContract, which asserts the upstream fact directly – that first_order returns a non-scalar for a constant response, and that its np.ptp(y) == 0 check does not fire for a near-constant one. Those hold on every NumPy version.

The general lesson: when a test’s failure mode depends on a dependency version, say so in the test, and add a second test that pins the underlying behaviour instead of the symptom.

Two tests the mutation check caught

Both were written to look correct, passed on the real implementation, and would have gone on passing after the guard they nominally covered was deleted.

An assertion satisfied by somebody else’s warning

TestARDBoundWarning originally asserted

assert "bound" in messages.lower()

Silencing MCPost’s ARD-bound warning left the test green, because sklearn emits its own ConvergenceWarning on the same fit:

The optimal value found for dimension 2 of parameter k1__k2__length_scale is close to the specified upper bound 100.0.

The substring "bound" was coming from sklearn, not from the code under test. The assertion now matches "pinned at the lower bound", a phrase only MCPost’s own message contains.

This is the same shape as the classic trap of asserting a result equals 3.0 when the fixture’s parameter is also 3.0: the assertion is true for a reason that has nothing to do with the property being tested.

A conditional assertion that quietly stopped asserting

The same test wrapped its check in if n_at_upper:. Had the fixture ever stopped pinning a length scale — a real possibility once the optimiser default changed — the assertion would have been skipped and the test would still have reported success. It now asserts n_at_upper > 0 unconditionally first, and fails with an explicit “pick a new fixture” message if the precondition dies.

A mutation aimed at code it could not reach

Entry 11 originally targeted TestSobolGFunction, which passes n_restarts_optimizer explicitly — so reverting the default could never affect it, and the green result meant nothing. It was retargeted at TestShippedDefaultsRecoverTruth, which passes no tuning at all.

Redundant code found

Mutation 3 stayed green, and correctly so. _holdout_target_is_degenerate had

if s_full == 0.0:      # the target itself is constant
    return True
return s_hold <= rtol * s_full

A globally constant target forces s_full == s_hold == 0, and 0 <= rtol * 0 is already True, so the early return could never change the answer. It was removed rather than left as unreachable reassurance.

Defensive-only code

Mutation 5 stayed green and the clip was kept anyway, with this recorded here so nobody mistakes the green for coverage.

The original bug report described _distance_correlation returning values far outside [0, 1] — “3e13” — on near-constant columns. That does not reproduce on the shipped v0.1.4 code. A search over 1200 adversarial input pairs (constants, near-constants at 1e-16 and 1e-30, two-valued columns, denormals at 5e-324, magnitudes at 1e308, single huge outliers, integer-valued columns; n ∈ {10, 25, 75, 150}) produced no value outside [0, 1] and no non-finite result. The maximum observed was exactly 1.0, for dCor(x, x).

Cauchy–Schwarz bounds the ratio by 1 analytically, and the existing max(..., 0.0) plus denom > 0 guards already handled the lower end, so no input the search could construct made the clip load-bearing. It stays as defence in depth against future changes to the centring step.

The search did surface a different, real defect in the same function — silent zeros from overflow/underflow at extreme input scales — which mutations 4 and 6 do cover.

Reproducing

bash scripts/mutation_check.sh