METHODOLOGY

How we measure and how we reach a conclusion.

The process is designed to make comparisons as fair and repeatable as practical. Below are the main principles without unnecessary technical detail.

01Consistent conditions

Key measurements are repeated more than once before and, where optimization is performed, after the changes. The goal is to keep both sides of the comparison under the same conditions.

Before each capture the system goes through a 10-minute idle normalization period, followed by a check of its state: background load, pending updates and signs of thermal throttling. If conditions are not suitable, the measurement is repeated later. In-game captures use the same title, settings and scene on both sides of the comparison.

02Evidence threshold

A number being different is not enough. A difference counts only when it clears two bars at once.

|Δ| ≥ 1.5 × pooled standard deviation of the runs AND |Δ%| ≥ the practical floor for that metric

The first bar asks whether the change is larger than the measurement's own run-to-run variation. The second asks whether it is large enough to matter in practice. A change that clears only one of them is reported as within noise.

This is an engineering repeatability threshold, not a formal statistical significance test, and the report never presents it as one. If a measurement is too unstable on either side (coefficient of variation above 5%), it is marked inconclusive. Some indirect metrics are shown only as advisory context.

03Result categories

IMPROVED
Improved: the change clears the evidence threshold in the better direction.
REGRESSED
Regressed: the change clears the evidence threshold in the worse direction. It stays in the report.
WITHIN NOISE
Within noise: the difference does not clear the evidence threshold, so no change is claimed.
NOISY / INCONCLUSIVE
Inconclusive: the measurement was too unstable to support a clear result.
ADVISORY
Advisory: information that helps describe the system but does not determine the final conclusion by itself.

CORE PRINCIPLE

"If the measurement does not support a clear conclusion, the report says so. No measurable difference is still a useful finding."

View a sample Verified Report

The public page explains the logic of the process. The per-metric floors are calibrated internally and are recorded in each report.

Want this process applied to your own system?

Start with a System Check or book a Full Optimization

MEASURE → DIAGNOSE → OPTIMIZE