A backtest is a measurement taken on data you have already seen. That is not a flaw to be apologised for — it is the only measurement available before capital is committed — but it does mean the number it produces is closer to an upper bound on expectation than to an estimate of it. Most of the disappointment that follows a live deployment is the distance between those two things being discovered slowly, and expensively.
The largest single contributor is multiple testing. A researcher who evaluates two hundred variations of a signal and reports the best one has not found a strategy with a Sharpe of 2.1; they have found the maximum of two hundred draws from a distribution whose true mean may well be zero. The correction is not mathematically difficult — record how many configurations were tried and deflate the reported statistic accordingly — but it requires the research process to track attempts honestly, which makes it a governance problem rather than a modelling one.
The second contributor is cost modelling. Strategies are frequently backtested against mid prices with a flat basis-point assumption for slippage. Both are optimistic. Crossing the spread is a real cost, queue position determines whether a passive order fills at all, and market impact grows with participation rate in a way a constant cannot capture. A signal with genuine predictive power and a holding period measured in minutes can be entirely consumed by the cost of acting on it.
Capacity is the third, and it is usually discovered last. A strategy tested at notional size behaves differently at the size that would justify running it. If the edge lives in the smallest and least liquid names, the allocation that makes the return meaningful is often the allocation that destroys it. Establishing the capacity curve before the pilot is considerably cheaper than establishing it afterwards.
The practical discipline is to treat the research result as a hypothesis carrying a stated confidence, then design the pilot as the test of it. Agree the out-of-sample window, the cost model and the decommissioning rule before the strategy goes live, and the conversation six months later is about evidence rather than about attribution.
