Verifiability

Off-band calibrations & methodology limits

The following profile-asset combinations sit outside their methodology-expected conformance bands as of the most recent snapshot. We list them here rather than quietly retrying calibration loops, because the deviations carry methodological information rather than being pure simulator deficiencies.

Empirical identifiability verification (BTC)

The strongest test of an FM definition is whether known historical stress events themselves satisfy it. Below: realized vol of BTC's 4 empirical stress anchors, under the same metric pipeline used for synthetic replicas.

Reading. All four BTC stress anchors fall below the 1.5× baseline threshold (1.20 absolute) required for VOL_EXPANSION classification, even though their absolute realized vols (0.45–0.92 annualised) and drawdowns (−26% to −66%) are unambiguously stress-grade. The methodology definition simply does not separate “BTC stress” from “BTC normal” given an 0.80 baseline. Our synthetic VOL_EXPANSION × BTC probe at 0% conformance is therefore a downstream symptom of the definitional limit, not a primary failure of the simulator.

Structural off-band combinations (5)

Conformance falls outside the expected band even when accounting for sampling uncertainty (Wilson 95% CI strictly below band). These are not closed by further calibration loops at fixed methodology thresholds.

Implication for robustness scores: These BTC and WTI/QQQ synthetic stress probes contribute to the spectrum-coverage of their failure-mode buckets but with reduced weight (per-bucket shrinkage when n_in_bucket < 10). The empirical historical anchors (Luna 2022, FTX 2022, BTC Crypto Winter 2018; WTI Saudi-War 2014, Negative-Pricing 2020) carry the primary information for these failure modes on these assets — the synthetic probes serve as auxiliary spectrum-fillers, not as primary evidence. For BTC specifically, this is forced by definitional limits of the FM gating (above), not by simulator deficiency.

Sampling-marginal off-band combinations (4)

For these combinations the Wilson 95% CI overlaps the expected band — we cannot statistically distinguish them from in-band conformance at n=50. Note that replicas within a profile-asset combination are not strictly i.i.d. (they share antecedent-phase parameters, calibration template, and observer biases), so the effective sample size is somewhat below 50 and the true CI is likely wider than reported. The Wilson bound here is therefore a lower bound on uncertainty.

Resolution path

• BTC × 3 setup-profiles (definitional limit): we do not retry calibration. The empirical-identifiability table above shows the FM gating is not separating for BTC at the uniform threshold. For BTC the trust-layer relies primarily on empirical historical anchors (Crypto Winter, Luna, FTX); synthetic probes provide auxiliary spectrum-coverage with reduced weight via shrinkage.
• WTI low_vol_grind + QQQ low_vol_grind (calibration limit): no further re-tuning at fixed thresholds. These probes contribute partial spectrum-coverage; empirical low-vol periods (SPY 2017, GOLD 2014) carry the primary VOL_COMPRESSION information.
• 4 sampling-marginal cases: re-evaluation on a future n=200 generation pass. If the Wilson CI tightens and no longer overlaps the expected band, we will reclassify as structural.
• Methodology revisions: only after anchor validation against historical stress events. Asset-relative threshold tuning is rejected because it would break definitional invariance, not because it would break comparability per se. The complementary percentile-of-own-vol diagnostic is on the roadmap as an additive disclosure, not a redefinition.

Edge case: unclassified replicas

In our current per-FM-bucket score (V2), replicas that fail every FM gating definition (sub-threshold) are excluded from the composite. This means the composite is conditioned on “identifiable regime” rather than on the full outcome distribution. We disclose this as a known bias toward stress-relevant outcomes; the planned correction is to assign sub-threshold replicas to a BASELINE bucket so they enter the composite at neutral weight. This is tracked alongside the V2 production activation work.

Methodology limitations & open issues

• Identifiability per asset. The FM gating definitions are not guaranteed to be separating across all assets (BTC is the demonstrated case). Identifiability is verified case-by-case via the empirical-anchor table.
• Power of test at n=50. With p̂ ≈ 0.5, the Wilson 95% CI half-width is ~14pp. Small effect sizes (≤14pp from band) are not statistically separable from sampling noise. Larger n is needed to resolve marginal cases definitively.
• Multiple testing. 32 datasets × 1–3 FM tags ≈ 60–80 implicit hypothesis tests. We do not apply multiplicity correction to the conformance bands. At α=0.05 nominal one would expect 3–4 false-positive off-band findings under no-effect; we report 9 off-band, of which 5 are structural and 4 sampling-marginal — roughly consistent with the expected false-positive rate among the marginal subset.
• Window-mixing in 6-month case slices. Stress events concentrated within a sub-window of the case (e.g. Volmageddon 2018 was a 1-day VIX spike inside a 6-month case window) get diluted in aggregate statistics. We chose 6-month windows for trading-strategy realism, not for FM identifiability — this is a conscious trade-off.
• CoT calibration drift. Per-asset observer weights and biases are derived from CFTC Commitment-of-Traders reports over 2006–2026. Calibration is point-in-time; we do not currently track whether the simulator's behaviour drifts as we add later CoT data.
• Anchor selection bias. The 31 empirical historical anchors were selected for stress severity and event-type coverage, not by random sampling of all 6-month windows. Selection favours canonical events (Lehman, Dotcom, COVID) over the full distribution of stress regimes that history actually produced.
• No out-of-sample walk-forward validation. Robustness scores are computed in-sample relative to the curated case catalog. Strategies are not validated on held-out post-catalog periods. This is on the roadmap but not yet implemented.
• Replica non-independence. Within a profile-asset combination, the 50 replicas share antecedent-phase parameters and calibration templates. Effective sample size is below 50 in any analysis that assumes i.i.d. replicas. Wilson CIs reported here are lower bounds on true uncertainty.
• Score definition-dependence. Any robustness score that aggregates over FM-classified replicas inherits a structural dependence on the FM definitions themselves: refining a definition (e.g. splitting SHARP_CRASH into mild and deep variants) can shift a strategy's score even if its trading behaviour is unchanged. This is a closed-loop risk for any FM-bucketed metric. The roadmap addresses it via two changes: (i) a two-step aggregation that conserves run-level mass before projecting onto FM-buckets, and (ii) a definition-independent score component (raw drawdown distribution across all replicas, no FM-bucketing) that anchors the composite to a non-classifier-derived quantity. Currently the production composite uses the v1 path (no FM-bucketing in composite), which sidesteps the issue at the cost of not exposing per-FM-bucket detail in the score itself; the per-FM-bucket detail is reported separately in the regime-performance breakdown.