Strategy Discovery Lab — Honest Backtest Search

Your best backtest is probably lying to you

Try enough strategy combinations and one will look brilliant — by luck alone. A
Sharpe of 1.8 means very little if it's the best of 500 you tried; it's the tail
of a random cloud, not an edge. Most "strategy finder" tools hide this and hand
you the flashiest curve. This one doesn't.

Strategy Discovery Lab scans your price data for candidate strategies, ranks
them into a leaderboard, and then tells you the truth about each one: the
Deflated Sharpe (corrected for how many you tried), whether the edge survived out
of sample, and the probability the whole thing is overfit. A great-looking curve
is presented as a hypothesis to test — never a strategy to trade.

Runs on your own CSV. Nothing is uploaded, and the output is a single
self-contained HTML report you open locally.

What you get

A leaderboard — the "you found a strategy!" moment

It backtests hundreds of classic setups across ten families — MA/EMA cross,
MACD, momentum, Donchian breakout, RSI / Stochastic / Bollinger reversion, a
trend filter, and a volatility-regime filter that only trades a trend in the
volatility state you pick — each over a grid of parameters. The best rise to the
top with equity curves.

The honest read — the part hype tools skip

• Deflated Sharpe Ratio — the Sharpe corrected for how many you tried. The
  best of 500 attempts is inflated; this removes the luck.
• Out-of-sample split — how much edge survives on data it was never ranked on.
• Overfit probability — a PBO-style estimate of how often the in-sample
  winners fall below the pack out of sample.
• The luck cloud — a histogram of every Sharpe you tried, with your best and
  the "pure-luck expected maximum" both marked, so you can see your winner sitting
  in the tail of a random distribution.
• A conservative, plain-English verdict: 🎲 likely noise / ⚠️ weak / 🤔 worth validating.

An overnight mode that gets more skeptical the longer it runs

Leave it scanning while you sleep. It keeps drawing random strategies, accumulates
every trial into a persistent ledger, and refreshes the report on an interval.
Because the honesty correction uses the cumulative number tried, the "pure luck"
bar climbs as the count grows — scanning 40,000 strategies overnight doesn't make
it easier to find an edge, it raises the bar a lucky curve must clear. Stop any
time with Ctrl+C; resume later and the count carries on.

python3 scripts/lab.py --prices my_prices.csv --watch --minutes 480 --out report.html

Mines here, validate there

This tool's job is to find candidates and give a light, honest reality check —
just enough not to lie to you. It deliberately does not run the full
statistical battery (bar-permutation MCPT, PBO/CSCV, SPA/StepM, stationary
bootstrap) — that is a separate, heavier job. Export the survivors and hand them
to a dedicated validator:

python3 scripts/lab.py --prices my_prices.csv --export survivors.csv
# survivors.csv is a per-bar returns matrix, ready for a candidate-matrix validator

Quickstart

# scan your own data (CSV needs a date column and a close column)
python3 scripts/lab.py --prices my_prices.csv --n 500 --out report.html

# try it on the bundled sample data, report in Chinese
python3 scripts/lab.py --demo --lang zh

# long-only (for markets you can't easily short), with a custom cost
python3 scripts/lab.py --prices my_prices.csv --long-only --cost-bps 2

Prerequisites: python3 with numpy and pandas. The honesty math uses the
standard library only — no scipy.

What ships

• Ten strategy families over parameter grids, plus a continuous random space for
  the overnight scanner.
• A vectorized, look-ahead-safe backtester (positions trade the next bar) with a
  per-turn cost and a --long-only mode.
• The honesty layer: Deflated Sharpe, out-of-sample split, overfit probability,
  the luck-cloud histogram, and a conservative verdict.
• A self-contained bilingual (English / 中文) HTML report — leaderboard, equity
  curves, DSR meter, and an honest "nothing survived" banner when a scan turns up
  only noise.
• Overnight --watch mode with a resumable ledger, and --export to hand
  survivors to a validator.
• A synthetic sample dataset, examples, and a passing test suite.

What this is not

Not a signal service and not a strategy — it contains no view on any market
and makes no performance claims; it reports what patterns did on your data.
Not a validator — it mines and reality-checks lightly, then hands off. Not a
promise of profit — most honest scans turn up nothing tradeable, and the report
says so.

FAQ

Does my data leave my machine? No. Everything runs locally; the report is a
file you open. Nothing is uploaded.

Will it find me a winning strategy? Usually not — and that's the point. On a
fixed set of classic patterns and one price series, most searches surface noise.
The tool's value is telling you that honestly, and flagging the rare candidate
worth taking to a full validation.

Which Python? 3.8+, with numpy and pandas. pytest is only needed for the tests.