Why a Model with No Predictive Power Can Score 90% Accuracy

2026-07-18 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, long

Summary

The article demonstrates how look-ahead bias can inflate model accuracy, specifically when outcome variables span multiple future observations and overlap between training and test sets. It simulates a market with no predictive edge, where daily returns are independently drawn from a bell curve centered on zero. A model predicting 20-day cumulative returns achieves 90.5% accuracy using a one-nearest-neighbor classifier and walk-forward validation with no gap. This inflated score arises because neighboring training and test labels share 19 of 20 underlying returns, leading to 91% agreement between adjacent labels. Applying a "purging" fix by introducing a 20-day gap between training and test sets in TimeSeriesSplit eliminates this overlap, reducing accuracy to 50.3%, which aligns with the expected 50% for data with no signal. The article highlights that this 40.2 percentage point overstatement is due to leakage.

Key takeaway

For Machine Learning Engineers building time-series models, if your outcome variable spans multiple future observations, you must implement purged cross-validation. Failing to introduce a sufficient gap between training and test sets will lead to look-ahead bias, artificially inflating your model's reported accuracy by potentially over 40 percentage points. Ensure your validation strategy prevents any training label from reaching into the test period to avoid deploying models that appear performant but lack true predictive power.

Key insights

Look-ahead bias from overlapping outcome windows can inflate model accuracy significantly, even with no predictive power.

Principles

• Outcome variables spanning future observations can create look-ahead bias.
• Neighboring time-series labels often share underlying data, leading to high agreement.
• Measured accuracy can be inflated by 40+ percentage points due to label overlap.

Method

To prevent look-ahead bias from overlapping outcome windows, introduce a "purging" gap (equal to the outcome horizon) between training and test sets in walk-forward validation.

In practice

• Use TimeSeriesSplit with a gap parameter.
• Set gap to the outcome horizon (e.g., 20 days).
• For variable horizons, apply exact purging.

Topics

• Look-ahead Bias
• Time Series Validation
• Purged Cross-Validation
• Model Accuracy
• Financial Machine Learning
• TimeSeriesSplit

Best for: Machine Learning Engineer, Data Scientist

Related on AIssential

• I Trained a Model on 150 Years of Football to Predict Today’s World Cup Final — Honest machine learning requires rigorous data handling and validation to avoid self-deception and ensure reproducible, reliable predictions.
• The Stone in the Lake: What I Learned Trying to Forecast London’s NO2 — Complex time series models do not automatically outperform simpler baselines, and stability across test periods is critical.
• ​​Time Series Cross-Validation: A Guide to Techniques & Practical Implementation — Time series cross-validation maintains chronological order for robust model evaluation and accurate forecasting.
• The first thing you need to understand when you start with AI — Effective machine learning models balance bias (underfitting) and variance (overfitting) to generalize well.
• Building Models in Two Worlds: From Latent Constructs to Behavioral Signals — Modeling latent constructs requires building variables from indirect evidence, while behavioral data uses ready-made signals, fundamentally altering modeling approaches.
• From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators — Cross-validation-tuned regularized estimators exhibit asymptotic risk equivalent to SURE-tuned shrinkage estimators in the normal means model.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.