METHODOLOGY
How a claim earns its verdict.
This is the public summary of the protocol behind every Alchemy report. The principles are open; the calibration internals stay in the lab.
01Controlled measurement
Every metric is executed in repeated runs per side — before and after — with warm-up runs that are excluded from the statistics. Both sides are measured under the same conditions: same power plan, same thermal state, network idle, same driver versions unless the driver change is itself the intervention.
Before anything is measured, environment gates check that the system is actually in a measurable state: background load, thermal throttling, pending updates. If a gate fails, the protocol says ABORT — meaning we don't measure. A measurement taken in a noisy environment isn't data; it's decoration.
02The confidence rule
A difference only counts as real when it clears both bars:
The first bar is statistical: the difference must exceed the measurement's own run-to-run noise. The second is practical: it must also exceed a calibrated, per-metric floor below which a difference is real on paper but meaningless in use.
Stability is checked too: metrics whose coefficient of variation is too high during measurement are marked unstable and cannot produce a verdict. And advisory proxies — indirect indicators like responsiveness windows — are reported for context but never counted in the final verdict.
03The five verdicts
-
IMPROVED
The change exceeded both the noise band of the measurement and the practical floor — a defensible improvement.
-
REGRESSED
The change made the metric measurably worse. Yes, we report that too — and it goes in the headline if it happens.
-
WITHIN NOISE
The difference is smaller than the measurement's own run-to-run variation. No claim can honestly be made.
-
NOISY / INCONCLUSIVE
The metric was too unstable during measurement (high variance) to support any conclusion in either direction.
-
ADVISORY
A proxy metric reported for context only — it never drives the final verdict.
HONESTY CLAUSE
"A result inside the noise band is reported as exactly that. "No defensible change" is a finding — not a failure."
The report shows the principles in action. Internal calibration thresholds are not published — the summary is public, the instrument is not.