Backtesting without fooling yourself

You've just run a backtest and the curve goes up and to the right. Nice. Now the uncomfortable question: did you find an edge, or did you build a strategy that happens to fit the last few seasons perfectly? A backtest replays history and tells you how a strategy would have done. Used well, it's the closest thing to proof an edge exists. Used badly, it's a machine for generating false confidence.

How overfitting sneaks in

Overfitting is when a strategy learns the specific quirks of your historical sample rather than a general truth. It's seductive because the symptom looks like success: a beautiful, steep P&L curve.

It creeps in through small decisions:

• Nudging an odds range until the curve improves.
• Adding a filter that only helps because it excludes three big historical losses.
• Trying twenty league combinations and keeping the best-looking one.

Each tweak fits the strategy tighter to this data, and further from the future.

The tell-tale signs

• Very few bets carrying a big yield.
• Many filters, each finely tuned.
• A flat curve except for one or two decisive runs.
• An edge you can't explain in one sentence.

Out-of-sample: the honest test

Never judge a strategy on the same data you built it with. Split your history:

1. In-sample (say, the older 70%): explore, form hypotheses, tune here.
2. Out-of-sample (the newer 30%): touch it once, at the end, to confirm.

If the edge holds out-of-sample, it's far more likely to be real. If it evaporates, you overfitted, and you just saved yourself from a live losing streak.

Practical rules of thumb

• Decide first. Choose your filters before looking at their individual P&L impact.
• Prefer stable. A slightly worse in-sample result that holds beats a spectacular one that's fragile.
• Presume guilt. Treat every extra filter as guilty until proven innocent.
• Trust the close. Once live, trust closing line value over short-term profit; it's the out-of-sample test that never stops running.

The discipline is uncomfortable: it makes your strategies look less impressive. That's exactly why it works.