A backtest is a simulation of how a strategy would have performed on historical data. Done well, it is the most powerful tool a quantitative researcher has - a time machine that lets you live through years of market history in seconds and learn what your idea would have done. Done poorly, it is the most reliable way ever invented to manufacture overconfident, money-losing systems. The gap between the two is not a matter of mathematical sophistication. It is a matter of discipline. This chapter is about that discipline. The unsettling truth: every working researcher has, at some point, produced a backtest that promised Sharpe 3.0 in development and delivered Sharpe 0.2 in live trading. The mathematical machinery did not lie - the researcher lied to themselves, in small, almost invisible ways, until the cumulative effect was a beautiful equity curve and a wretched out-of-sample reality.
Figure 1. Concept map for this episode: fight, desk translation, what to lock.
What a good backtest must do
A defensible backtest must satisfy seven properties: 1. Point-in-time data. At every historical decision point, the model must use only data that was actually available at that time. No look-ahead. 2. Survivorship-free universe. The asset universe at each historical point in time must include every name that was actually tradable then, including names that subsequently delisted, went bankrupt, or were acquired.
The seven cardinal sins of backtesting
Each sin below has destroyed real money. Each sin is easy to commit and easy to overlook. The discipline of naming the sin before deploying any backtest result is the single best protection a researcher has. Sin 1 - Look-ahead bias. Using future information to make a past decision. The most common form is using same-day close to make a same-day open decision. A subtler form: using a feature computed from full-month data to decide a trade made on the second of the month. The fix: shift every feature back by at least one period, and spot-check by manually inspecting a handful of historical decisions against exactly what data would have been available at that moment.
The minimum acceptable backtest
A backtest that does not pass all seven properties of §32.1 is not deployable. The minimum acceptable backtest: - Includes the full historical universe with point-in-time membership. - Uses lagged features (today's feature value predicts tomorrow's return, never today's return). - Applies friction from a measured friction governor. - Uses limit fills only when the market actually crosses the limit price; market orders fill at the next available price including spread. - Splits data into training and out-of-sample validation periods that never overlap.
Definition
A data set is point-in-time when, for any historical date, the values it stores for that date are exactly what would have been observable on that date - not revised values, not later corrections, not anything that incorporates information from after the fact. The opposite is a revised or snapshot data set: it contains values as they appear today, including all subsequent restatements, corrections, and re-classifications. Almost all freely available financial data is the revised kind. Building a point-in-time data set is expensive and tedious - and absolutely necessary for honest backtesting.
Worked picture
You are building a strategy that uses 20-day average volume as a feature. On 1 March 2025, the model trades a stock based on its average volume "over the last 20 days." Two reasonable definitions: - Definition A: average volume over the 20 trading days ending 28 February 2025. (Correct: only uses data available at start-of-day 1 March.) - Definition B: average volume over the 20 trading days centred on 1 March 2025 (10 before, 10 after). (Catastrophic: uses 10 future days of data to "predict" a trade taken at the start of 1 March.) Definition B produces a backtest that looks brilliant. It is also a complete fiction. The strategy in production cannot use Definition B - the future ten days have not happened yet. Backtest performance will collapse the moment the strategy goes live. The discipline: for every feature, write down the latest timestamp the feature could possibly observe to be honest. Verify that this timestamp is strictly earlier than the decision timestamp. Repeat for every feature. Boring, slow, irreplaceable.
What breaks in production
The most insidious form of overfitting is not deliberate. The researcher does not say "I will try 100 strategies and report the best." Instead, they try one strategy, find it slightly disappointing, tweak the entry rule and try again, find that better, tweak the stop and try again, swap the universe and try again. After two weeks of "iteration," they have, in effect, run 50 backtests on the same data - but the final reported number is presented as if it were the result of a single confirmatory test. This is the garden of forking paths, named by the statistician Andrew Gelman. The cure is to log every decision you ever made on the data, count them honestly, and apply the same multiple-testing correction to the implicit-search space as you would to an explicit one. In practice this means accepting that any "best Sharpe found through iteration" is roughly half of what the raw number says, and demanding correspondingly more confirmatory out-of-sample evidence before trusting it.
Key idea
A backtest is not a proof that a strategy will work. A passing backtest only means the strategy has not yet been ruled out. The seven properties of §32.1 are the minimum bar to clear before the question "does this work?" even becomes meaningful.