The bootstrap that would not reproduce

Two runs of the same analysis — same data, same workflow, same seed — return two different confidence intervals. Not dramatically different: [12.31, 13.02] against [12.32, 13.01]. A wobble in the second decimal, the kind of thing a tired person files under “bootstrap things” and moves past. Don't. This is the walk that turns “flaky” from a mood into a diagnosis — and the culprit, as it usually is, sits in code you wrote yourself.

Two intervals, one seed#

The setting is ordinary statistics. A right-skewed, positive measurement column: 431 rows delivered by an upstream concatenation that duplicated 19 of them, so 412 rows after deduplication. Skew makes mean ± standard error a dishonest summary, so the estimand is the median: 12.67, the average of the 206th and 207th order statistics. For an interval, the obvious nonparametric tool: the bootstrap. Draw 412 rows with replacement from the 412, take the median, repeat B = 10,000 times; each resample omits about 36.8 % of the rows on average — the e⁻¹ of folklore — and the spread of the ten thousand medians estimates the sampling noise of yours. The 95 % percentile interval reads off the empirical 2.5th and 97.5th percentiles: in effect the 250th and the 9,751st of the sorted bootstrap medians. Exactly which quantile convention is the quantile op's declared business — anyone who has reconciled one library's “type 7” against another's default knows that footnote decides second decimals. There are finer intervals, BCa among them; for a plain median at n = 412, percentile is defensible.

So: seed 42, B = 10,000, [12.31, 13.02]. Run it once more before committing — out of tidiness, not suspicion. Same seed. [12.32, 13.01].

The statistician's reflex#

The honest first instinct is to quantify. The endpoints of a percentile interval are themselves Monte-Carlo estimates, with standard error √(α(1−α)/B) divided by the bootstrap density at that quantile. If the ten thousand medians have a standard deviation of about 0.18, then under a normal approximation the density near the 2.5 % quantile is about 0.058/0.18 ≈ 0.32, and the Monte-Carlo standard error of each endpoint is near 0.005. Both observed shifts are about 0.01 — two Monte-Carlo standard errors. Between runs with different seeds, that is shrug territory; if it bothers you, quadrupling B halves it.

But the seeds are not different. Under a fixed seed, the reference distribution of the difference is not “small” — it is a point mass at zero. The size of the discrepancy carries no information; its existence is the entire finding. What makes it all feel innocent is the other clue: the point estimate reproduces exactly, 12.67 in both runs — a full-sample median does not care about row order. Everything looks healthy except the interval's tails, and tails are precisely where a tired person expects noise to live.

Three suspects#

Suspect one: the GPU. WebGPU float execution is non-associative — reduction order is not guaranteed across schedules — which is exactly why the prod determinism tier promises reproduction within a declared tolerance class rather than bitwise equality. But a bootstrap index is an integer; there is no tolerance ball around “row 227”. A median is a selection — at even n, one addition and one halving — not a long accumulation. And 0.01 on 12.3 is about 8 × 10⁻⁴ relative, orders of magnitude beyond anything a selection op could declare with a straight face. Plausible suspect, weak motive.

Suspect two: the platform's random numbers. Scellis RNG is counter-based, Philox-style: a draw is a pure function of (seed, stream, counter) — no hidden global state, the same output for the same coordinates on any backend, with named sub-streams for shuffling, dropout, augmentation, and the state checkpointed with the run. Built to be innocent — but the doctrine of this platform is that innocence gets verified, not asserted. Suspect three: your own authored preprocessing block.

The first instrument is the tier ladder itself. A run's determinism tier is explicit — recorded on the run, never silently defaulted. Flip the workflow from prod to debug, which promises bitwise-identical replay for the same seed on the same device and pays for it honestly with deterministic kernel variants and fixed reduction orders (slower, and the contract says so). Two debug runs, two different artifact hashes. That is a verdict, not a symptom. At debug, “differs at all” is a defect by definition — no statistics required, no judgment call. Genuine entropy is entering the run.

The second instrument is the reference path. Every built-in numeric op carries a CPU reference — the one every GPU kernel is checked against. Run the workflow twice more there: bit-exact, independent of any GPU. Still two different hashes. The silicon stands acquitted; whatever is rolling dice lives in the program, not the hardware.

Bisection by hash#

Here the platform stops being an instrument and becomes a map. Every intermediate artifact is content-addressed, and a run's provenance records the hash at every node. Lay the two debug runs side by side and walk the graph: ingest — identical hashes. Typing, summaries — identical. The deduplication block — different. First divergence; everything downstream inherits the poison. The node is user/kemal/dedup_rows@1: yours, authored in an afternoon some weeks earlier.

// user/kemal/dedup_rows@1 — reference body (the guilty version)
const rows = dedupeByKey(input.rows, "sample_key");
// notebook reflex: "shuffle so later splits don't inherit file order"
for (let i = rows.length - 1; i > 0; i--) {
  const j = Math.floor(Math.random() * (i + 1)); // <- unseeded entropy
  [rows[i], rows[j]] = [rows[j], rows[i]];
}
return { rows };

The deduplication is the honest half. After it comes a Fisher–Yates shuffle over the rows, driven by Math.random() — unseeded, unnamed, invisible entropy. A notebook reflex: shuffle after dedup so later splits don't inherit file order. You had forgotten it was there. The block's metadata, meanwhile, declares it deterministic — a declaration the platform provisionally takes at your word for a sandboxed body on your own device, and it is honest about that. Static analysis does not convict this kind of lie; the contract's job is different. It makes the lie falsifiable in a single evening.

The fix deletes itself#

The most instructive part is what the shuffle was for: nothing. A bootstrap draws rows by uniform random indices, and a permutation composed with uniform draws is still uniform draws. Both runs' intervals were valid 95 % percentile intervals, drawn from exactly the same Monte-Carlo distribution. No bias anywhere; nothing wrong with either number.

So version 2 of the block simply deletes the shuffle — keep-first deduplication over identical input bytes is order-stable on its own. Where a shuffle genuinely matters, a holdout split say, you draw it from the run's seeded stream — a named sub-stream of the counter-based RNG — and it reproduces and checkpoints like everything else. Version 1 does not vanish, either: its bytes are pinned by hash in the provenance of both crime-scene runs, citable forever. You cannot quietly rewrite your mistake; you can only supersede it. Immutability is unflattering, and it is worth making peace with.

tier debug · seed 42 · same device
  run A  -> ci_artifact  sha256:6b01…e4
  run B  -> ci_artifact  sha256:9d57…22   ✗ debug promises bitwise equality

after the fix — user/kemal/dedup_rows@2
  run A′ -> ci_artifact  sha256:c8a3…7f
  run B′ -> ci_artifact  sha256:c8a3…7f   ✓ identical bytes

Then the real run: B = 10,000, seed 42, tier debug — [12.31, 13.02]. The same digits as the first run of the evening, and for the first time deserving the definite article: the interval.

What debug actually pins#

The thing to keep from this walk is the ladder, stated precisely. debug pins bitwise-identical replay — same seed, same device — via deterministic kernels, fixed reduction orders, and the counter-based RNG, at a real throughput cost. It does not promise bitwise equality across devices: floats on different GPUs legitimately disagree, so cross-device bit-exactness is the reference path's job, and prod promises a declared tolerance class instead. fast permits nondeterminism only where it is declared safe. And no tier is ever silently defaulted — it is stamped into the run's provenance and travels with the link you share.

Commit the workflow and mint the snapshot link:

scellis.com/studio?repro=c8a3…7f
  dataset@1 · user/kemal/dedup_rows@2 · B 10000 · seed 42 · tier debug

Anyone who opens it re-resolves the committed bytes — data, block versions, seed, tier; “bit-for-bit” means exactly those bytes. Re-executing on the reference path reproduces them exactly; re-executing on another GPU at prod reproduces within the declared tolerance — and the platform tells you which of the two you got, rather than promising more than floats can keep. The practice, step by step, is in the guide.

“Flaky” is not a diagnosis. It is a confession that your system holds entropy you have not named. Statistics trains us to quantify the uncertainty we mean to have; the least we can ask of our tools is that any uncertainty they add comes with a name, a stream, and a seed.