The Illusion of Control: Why Your Backtested Maximum Drawdown is a Lie

The Seductive Allure of the Backtest

For the quantitative trader, the backtest is the ultimate source of truth. It is a window into the past, a way to see how a strategy would have performed in a variety of different market conditions. And one of the most important outputs of any backtest is the maximum drawdown. It is the number that tells you the worst-case scenario, the maximum pain that you would have had to endure to reap the rewards of your strategy.

But what if that number is a lie? What if your backtested maximum drawdown is not a worst-case scenario, but rather a best-case scenario? This is the uncomfortable truth that many quantitative traders are unwilling to face. Your backtested maximum drawdown is almost certainly a significant underestimate of what you will experience in live trading.

This article will explore the various biases and pitfalls of backtesting that lead to this underestimation of drawdown risk. It will also provide a framework for developing more realistic drawdown expectations.

The Seven Deadly Sins of Backtesting

There are seven key reasons why your backtested maximum drawdown is likely to be a lie:

1. Look-Ahead Bias: Look-ahead bias is the most insidious of all backtesting biases. It occurs when your backtest uses information that would not have been available at the time of the trade. For example, if you are using accounting data to make trading decisions, you need to be sure that you are using the data that was available at the time of the trade, not the data that was released later.

2. Survivorship Bias: Survivorship bias is the tendency to only include the surviving assets in your backtest. For example, if you are backtesting a strategy on the S&P 500, you need to be sure that you are using the historical constituents of the index, not the current constituents. The companies that have been removed from the index over time are likely to have been poor performers, and by excluding them from your backtest, you are artificially inflating your returns and underestimating your drawdowns.

3. Transaction Costs: Most backtests do not accurately account for transaction costs. They either ignore them completely or they use a generic, one-size-fits-all estimate. In reality, transaction costs can have a significant impact on the performance of a strategy, especially a high-frequency strategy. And these costs will be most significant during periods of high volatility, which is exactly when you are most likely to be in a drawdown.

4. Slippage: Slippage is the difference between the price at which you expect to execute a trade and the price at which you actually execute it. It is a fact of life in live trading, but it is often ignored in backtesting. And like transaction costs, slippage is likely to be most significant during periods of high volatility.

5. Data Errors: The historical data that is used for backtesting is often riddled with errors. These errors can be difficult to detect, and they can have a significant impact on the results of a backtest. A single data error can be the difference between a profitable strategy and an unprofitable one.

6. Over-Fitting: Over-fitting is the practice of creating a strategy that is perfectly optimized to the historical data but that has no predictive power. It is the quantitative equivalent of driving by looking in the rearview mirror. And it is one of the main reasons why so many backtested strategies fail in live trading.

7. The Non-Stationarity of Markets: The final and most fundamental reason why your backtested maximum drawdown is a lie is that markets are not stationary. The statistical properties of markets are constantly changing. The future will not be like the past. And this means that your backtested maximum drawdown is, at best, a rough guide to what you might experience in the future.

A Framework for More Realistic Drawdown Expectations

Given the limitations of backtesting, how can we develop more realistic drawdown expectations?

1. Be a Skeptic: The first step is to be a healthy skeptic of your backtest results. Do not take them at face value. Assume that they are wrong, and then try to figure out by how much.

2. Add a Margin of Safety: A good rule of thumb is to add a margin of safety to your backtested maximum drawdown. A 50% margin of safety is a good starting point. So if your backtest shows a maximum drawdown of 20%, you should assume that you could experience a drawdown of 30% in live trading.

3. Use Monte Carlo Simulation: As we have discussed in a previous article, Monte Carlo simulation is a effective tool for developing more realistic drawdown expectations. By running thousands of simulations, you can get a much better sense of the range of possible outcomes.

4. Stress Test Your Strategy: Stress testing is another important tool for developing more realistic drawdown expectations. By subjecting your strategy to a variety of extreme scenarios, you can get a better sense of how it will perform in a crisis.

Conclusion: The Backtest as a Guide, Not a Guarantee

The backtest is a valuable tool, but it is not a crystal ball. It is a guide, not a guarantee. By understanding the limitations of backtesting and by using a variety of other tools to develop more realistic drawdown expectations, you can avoid the illusion of control and can be better prepared for the inevitable drawdowns that you will experience in live trading.