Your Backtest Is Lying to You — 7 Ways It Flatters You

2026-07-12 · Source: Machine Learning on Medium · Field: Finance & Economics — Capital Markets & Investment Management, FinTech & Digital Financial Services · Depth: Expert, long

Summary

This article details seven common backtesting mistakes that systematically flatter trading strategies, making them appear more profitable than they are in reality. These errors, ranging from widely known issues like overfitting and lookahead bias to more insidious problems, structurally remove friction, hide risk, or borrow unavailable information. Specific pitfalls include tuning parameters until the backtest "sings," using future data or only surviving instruments, ignoring trading costs like commissions and bid-ask spreads, and testing on in-sample data. Furthermore, the article highlights optimizing for average returns instead of drawdown, assuming clean fills at quoted prices, and using data resolutions too coarse to capture true market dynamics like queue position and slippage. Each mistake biases results positively, emphasizing the need for suspicion towards overly optimistic backtests.

Key takeaway

For quantitative traders and research scientists developing systematic strategies, you must approach backtest results with inherent suspicion. Your backtest is likely overstating profitability and understating risk due to structural biases. Prioritize rigorous validation by honestly modeling costs, using truly out-of-sample data, and evaluating strategies based on drawdown resilience rather than just average returns. This proactive skepticism will help you avoid costly live trading surprises and build more robust, survivable systems.

Key insights

Backtesting errors systematically flatter strategies, making them appear better than real-world performance.

Principles

• Fragility indicates overfitting, not a robust edge.
• Judge strategies by return per unit of drawdown.
• Coarse data cannot reveal granular market truths.

Method

To validate a backtest, sweep parameters for sensitivity, model trading costs honestly, guard out-of-sample data, and reconstruct realistic fills from order-level data.

In practice

• Sweep parameters to check for fragility.
• Model commissions and bid-ask spreads.
• Use order-level data for realistic fills.

Topics

• Backtesting
• Systematic Trading
• Overfitting
• Lookahead Bias
• Slippage
• Drawdown
• Trading Costs

Best for: Data Scientist, Research Scientist, Consultant

Related on AIssential

• Stop Testing One Indicator at a Time. Loop Your Way to Better Trading — Iterative "champion loops" overcome single-indicator testing biases to find optimal trading strategies.
• I Trained a Machine-Learning Model to Predict the Market. A Coin That Always Says “Up” Beat It. — Machine learning models often fail to find a significant, exploitable edge in predicting broad market direction from historical price data.
• Agentic AI Trading On Hyperliquid For Beginners (Codex 5.5) — AI agentic trading requires rigorous backtesting and optimization to avoid overfitting and identify truly robust strategies.
• How to estimate whether a trading strategy is profitable before you risk real capital — Long-term trading profitability hinges on the combined effect of win rate, risk-reward ratio, and risk per trade.
• How I Built an Institutional-Grade Quant Trading System as a College Student — Institutional-grade quant trading demands rigorous engineering discipline, decoupling, and advanced statistical validation beyond simple models.
• BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting — BacktestBench provides a critical benchmark for evaluating LLMs in automated quantitative trading strategy backtesting.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.