# Anti-overfitting: enforcement map The ten practices come from Hugo Fernandes (2026-11-03), written with Claude's help; Claude ran the simulations behind practice 10. See [credits](credits.md). Hugo updated practices 2 and 5 on 2026-11-07, and both are folded in below. Practice 2: use the Student t quantile rather than 1.96 when measurements are few, and compare candidates on matched (paired) seeds, which made his own intervals 3 to 10 times narrower; the noise that mattered most came from the evaluation windows rather than the seeds. Practice 5: measure ensemble diversity on the forecasts (correlation of errors, or disagreement on unlabeled data) rather than through subgroup performance, which reuses the labels, and keep equal weights fixed before anyone looks at a validation score. The status column rates the **default version-1 gate and ordinary harness**. Version 2 requires explicit opt-in; installing it does not migrate existing receipts or lanes. Its harness and gate checks are listed separately below. Declarations cannot prove training behavior, and neither policy creates OS-level read isolation. See the [v2 guide](../harness/docs/selection.md) for schemas, CLI commands, defaults and enforcement boundaries. | Practice | Default status and evidence | Opt-in v2 enforcement and limits | |---|---|---| | 1. Make validation mimic the test, using group/time splits. | PARTLY. The [harness freeze](../harness/idea_policy.py#L102) binds the holdout disposition and pass line, without a generic split algorithm. | The [gate schema](../selection/selection_gate/policy.py#L45) requires split, rationale and fold-local preprocessing declarations; random splits also require affirmative shared-unit evidence. The [harness](../harness/idea_policy.py#L95) hashes these into the freeze and checks them on later revisions. It cannot prove the split matches test sampling. | | 2. Average seeds and penalize instability using SEM, with the t quantile and matched seeds. | DONE. The [default gate](../selection/selection_gate/__init__.py#L38) requires at least two finite scores, uses sample SD and SEM, and accepts the highest mean only when mean − SD√(2 ln finalists) − 2 SEM exceeds the floor. The SEM multiplier is configurable; seed IDs need not represent independent units. V1 keeps the fixed multiplier: the t quantile is a v2 change. | The gate uses the [two-sided 95% t quantile](../selection/selection_gate/v2.py#L30) on df = n−1 for the margin, the lower bound and the submission ranking, never 1.96, and never narrower than it. It requires ten declared independent measurements by default, matching baseline/candidate IDs, distinct seeds or backtest splits, and a reasoned waiver below ten; a waiver to two measurements now pays a 12.706 multiplier rather than 2. [`paired_diff`](../selection/selection_gate/v2.py#L44) compares on matched measurement ids: supplying `baseline_sealed_scores` makes beating the baseline on the matched seeds an acceptance condition, and each finalist's paired interval against the leader is recorded with `best_separated_from_runner_up`. The [harness](../harness/selection_policy.py#L94) binds the minimum to lane limits at freeze and lint. Physical independence remains an operator responsibility, the paired baseline is opt-in rather than required, and nothing checks that the measurement schedule spans enough evaluation windows, which is where Hugo found the dominant noise. | | 3. Augment only training folds. | NOT DONE. The [default gate](../selection/selection_gate/__init__.py#L38) only consumes scores. | The [schema](../selection/selection_gate/policy.py#L56) requires training-only, label-preserving augmentation declarations. [Result admission](../harness/idea_policy.py#L155) and [config checks](../harness/selection_policy.py#L28), also used by lint, reject undeclared use and missing augmentation flags. They do not inspect arbitrary external transformations. | | 4. Take care when retraining on all data. | NOT DONE. The [default gate](../selection/selection_gate/__init__.py#L17) has no retraining inputs. | The [gate schema](../selection/selection_gate/policy.py#L64) requires a final-fit method; epoch retraining needs paired backtests with no observed split deterioration. The [harness freeze](../harness/idea_policy.py#L102) validates and binds this evidence. Neither component performs or verifies the actual retraining experiment. | | 5. Ensemble diverse models, with diversity measured on the forecasts. | NOT DONE. The [default gate](../selection/selection_gate/__init__.py#L53) selects an individual candidate. | The [ensemble helper](../selection/selection_gate/v2.py#L237) averages aligned predictions before scoring and counts the ensemble toward finalists; fitted weights are rejected, so the equal weights are fixed before any validation score is seen, and the row records `weights_fixed_before_scores`. [`forecast_diversity`](../selection/selection_gate/v2.py#L187) measures diversity from the members' own forecasts: mean pairwise disagreement, which needs no labels, plus mean pairwise **error** correlation when `sealed_labels` are supplied. Declaring diversity is refused, and so is arguing it from `subgroup_scores`, because slicing by subgroup reuses the labels. The [harness adapter](../harness/selection_policy.py#L53) supports additive per-row contributions. Nonlinear metrics need the Python scorer callback. Diversity is reported, not a threshold: no calibrated bar exists, so nothing is refused for being too similar. | | 6. Successive backtests and learning curves. | NOT DONE. The [default gate](../selection/selection_gate/__init__.py#L17) has no such requirement. | The [gate](../selection/selection_gate/v2.py#L151) requires expanding windows with disjoint held-out units, and [evaluation](../selection/selection_gate/v2.py#L114) requires a strict majority of wins by the configured margin and reports the worst split. The [harness](../harness/selection_policy.py#L104) replays the evidence during lint. Learning curves remain unenforced. | | 7. Beat a baseline using bootstrap over independent units. | PARTLY. The [default strict lower bound](../selection/selection_gate/__init__.py#L57) must beat the floor, without an independent-unit bootstrap. | The [bootstrap](../selection/selection_gate/v2.py#L69) groups paired differences within declared units and resamples unit means, requiring a positive 95% CI **and** the existing bound **and** backtests. The [harness audit](../harness/selection_policy.py#L104) reconstructs complete decisions. Real independence remains the caller's responsibility. | | 8. Count every trial; rerun the best few on fresh seeds/splits. | PARTLY. The [default gate](../selection/selection_gate/__init__.py#L59) records historical count and search optimism, without applying that penalty or enforcing a trial ledger/fresh reruns. | The [harness trial command](../harness/selection_policy.py#L40) appends completed/pruned trials; [audit](../harness/selection_policy.py#L104) rejects removal/replacement. The [gate](../selection/selection_gate/v2.py#L320) validates all recorded trials, best 3–5 completed candidates and fresh disjoint seed/split IDs. It reports mean − max(search optimism, measured prior) as expected score; this is not the eligibility threshold. Historical prior prefixes are anchored and replayed. Unregistered external trials cannot be discovered. | | 9. Keep a fixed hidden set and use it sparingly. | PARTLY. [Receipt creation](../selection/selection_gate/__init__.py#L65) is exclusive, but default `decide` neither scores data nor limits calls or finalist count. | The [pre-scoring wrapper](../selection/selection_gate/v2.py#L466) reserves budget before the scorer, including failures; the [ledger](../selection/selection_gate/policy.py#L146) binds set, competition, scoring ID and immutable lane budget. [Harness scoring](../harness/selection_policy.py#L53) logs a supplied batch; [lint](../harness/selection_policy.py#L104) checks budgets, hashes, history and anchors. The CLI consumes already computed scores, so cannot prevent their earlier computation. Storage protection and data isolation are external responsibilities. | | 10. Choose among several submissions. | NOT APPLICABLE (default). The [default gate](../selection/selection_gate/__init__.py#L61) returns one candidate or floor and never uploads. | The [gate](../selection/selection_gate/v2.py#L419) ranks eligible candidates by mean − t·SEM (practice 2's update; it was 1.96 before 11-07, which flattered the noisiest candidate) and returns up to the declared submission count. The [harness](../harness/selection_policy.py#L94) binds lane limits; [audit](../harness/selection_policy.py#L104) requires an accepted decision for a v2 shipped idea. The separate advisory [best-of helper](../selection/selection_gate/v2.py#L491) requires `current_best` on the sealed score scale, preferably derived from `current_sealed_scores`; public board numbers may be uncalibrated. It is unused by the toy/default gate. | Version 1 has no three-finalist cap. Exclusive receipt creation does not enforce one-time hidden scoring. Calling v2 `decide` with already computed scores cannot police their earlier computation either; `score_and_decide` supplies the optional pre-scoring boundary. Local hash chains and history anchors detect tampering relative to retained evidence, rather than authenticating an external auditor. Coverage: [harness integration](../harness/tests/test_selection_policy.py), [gate v2](../selection/tests/test_v2.py), [default gate](../selection/tests/test_gate.py), and [best-of scale contract](../selection/tests/test_best_of.py). Legacy replay coverage uses synthetic scores; no live-lane receipts or results are shipped. Run the full suite from the repo root with `python3 -m pytest -q`.