Strategy Edge Validator — Backtest Detector
Is your backtest a real edge — or an expensive illusion?
A profitable equity curve is not proof of an edge. Most retail strategies that "work" in backtest are overfit: the Sharpe is the best of many configurations you tried, an artifact of the stop/take-profit logic, or a lucky window. Trade one live and it quietly bleeds.
Strategy Edge Validator runs the statistical battery that separates a real signal from data-mined noise — entirely on your own return series or equity curve. No market data feed. No API keys. No strategy logic revealed. Your numbers never leave your machine.
What it checks (7 gates)
| Gate | The question it answers |
|---|---|
| Stationary bootstrap | Is the curve distinguishable from a zero-edge random series (autocorrelation-aware)? |
| Deflated Sharpe Ratio | You scanned N configs and kept the best — how much of the Sharpe is selection luck? |
| PBO / CSCV | Does the in-sample-best parameter stay best out of sample, or collapse? |
| Hansen SPA | Across a basket, does the winner really beat the benchmark once data-snooping is controlled? |
| Romano-Wolf StepM | Which candidates survive family-wise error correction? |
| Bar-permutation MCPT | Is the profit the signal — or just the exit path (brackets/stops)? |
| Out-of-sample split | How much edge survives the holdout? |
Validation is a router, not one test: different strategies die at different gates. Passing one proves nothing about the others.
Quickstart
```bash
pip install numpy scipy arch # arch is optional (only for SPA/StepM)
Single equity curve (one return per row, or a return column)
python3 scripts/cli.py --returns my_backtest.csv
Have an equity/NAV curve instead? It's auto-differenced
python3 scripts/cli.py --equity my_nav.csv
Deflate for how many configs you tried
python3 scripts/cli.py --returns my_backtest.csv --trial-sharpes all_trials.csv
A basket of candidates (PBO + SPA/StepM)
python3 scripts/cli.py --matrix candidates.csv --benchmark bench.csv
Bar-permutation MCPT (aligned market-return + position columns)
python3 scripts/cli.py --mcpt bars.csv --market-col mkt_ret --position-col pos
Annualized Sharpe + machine-readable JSON for pipelines
python3 scripts/cli.py --returns my_backtest.csv --periods-per-year 252 --json
```
Example output
STRATEGY EDGE VALIDATOR — overfitting battery [PBO OK] pbo=0.041 IS-best median OOS quantile=1.000 (M=8 strategies, 12870 combinations) [SPA beats benchmark] p_consistent=0.0088 [StepM] candidates beating benchmark (FWER<=5%): [4] VERDICT: LIKELY REAL EDGE (2/2 gates passed)
How to read the verdict
- All gates pass — the strongest historical evidence available. Still forward-test before risking real capital.
- DSR/PBO pass but MCPT fails — the edge is in your exit logic, not the signal. It will not generalize.
- SPA/StepM: no survivors — the "best rule" is data-mined. Don't trade any of them.
- Any single gate fails — NO-GO, or a downgrade to forward-testing.
What it does not do
- It does not model transaction costs, slippage, capacity, or live execution — clear those separately.
- Statistical passing raises the bar; it does not replace out-of-sample forward performance.
- The MCPT is the fixed-position variant and cannot detect same-bar look-ahead — audit that with a lag-1 signal check.
Full method math and known limitations ship inside (references/methods.md), and the tool ships with its own test suite (tests/) — a validator you can validate.
FAQ
Do I need to share my strategy? No. It runs on your return/equity numbers only. Nothing is uploaded.
What input format? A CSV of per-period returns, an equity/NAV curve, a candidate matrix, or aligned market/position columns. Samples included.
Sources? Bailey & López de Prado (Deflated Sharpe, PBO), Hansen (SPA), Romano-Wolf (StepM), Politis-Romano (bootstrap), Masters (permutation testing). All cited.
