11.6 Backtesting and Simulation: Procedures, Metrics, Biases & Scenario Analysis

11.6 Backtesting and Simulation: Procedures, Metrics, Biases & Scenario Analysis

Blueprint note: Backtesting and Simulation is a full Level II Portfolio Management learning module with eight learning outcomes. Vignettes describe a quantitative researcher's testing process and ask what is wrong with it, or present backtest output and ask what conclusion it supports.

---

1. Objectives of Backtesting

Backtesting approximates the real-life investment process of a strategy using historical data, to learn how it would have performed and, more importantly, what risks it carries.

Legitimate objectives:

• assess the risk-return profile of a strategy before committing capital;
• understand which environments the strategy performs well and badly in;
• estimate turnover, capacity, and transaction costs;
• test the sensitivity of the result to parameter choices;
• compare candidate strategies on a consistent basis.

What backtesting cannot do:

• prove that a strategy will work in future, since it reports one realisation of history;
• capture structural breaks — a regime that has not occurred in the sample is not in the test;
• reflect the market impact of the strategy's own trading once it is scaled up.

The core assumption is that the future resembles the past. Where that assumption is weakest — after a regulatory change, a structural shift in market microstructure, or a change in monetary regime — backtest results are least informative.

---

2. The Backtesting Procedure

Backtesting is not a single pass over history. It is a rolling-window procedure repeated period after period:

Step 1 — Strategy design. Specify the investment hypothesis, the universe, the ranking or signal rules, position sizing, the rebalancing frequency, and the transaction cost assumption. Everything must be fixed before the test runs.

Step 2 — Historical investment simulation. For each period:

• use a formation period ending at time $t$ to estimate parameters and rank securities, using only information available at time $t$;
• construct the portfolio at time $t$;
• hold it over the holding period to $t+1$ and record the realised return;
• roll the window forward one period and repeat.

The rolling window is what makes the test honest. Estimating a factor loading on the full sample and then "testing" the strategy over that same sample uses information from the future, which is look-ahead bias in its purest form.

Step 3 — Output analysis. Compute the distribution of realised returns and the risk metrics below, then examine the visuals.

Metrics reported

Visuals that carry information

• Cumulative return chart — reveals whether performance came from a few short episodes or accumulated steadily. A strategy whose entire excess return arrived in two months of one year is not a strategy.
• Rolling excess return or rolling Sharpe — shows whether the edge has decayed over time, which is common after a strategy is published.
• Drawdown chart — shows the depth and, critically, the duration of drawdowns.
• Return distribution histogram — reveals skew and fat tails directly.

---

3. The Three Structural Biases

Survivorship bias

The sample includes only entities that exist today. Funds that closed and companies that were delisted or went bankrupt are absent, so the measured return of the universe is biased upward and the measured risk downward. It is most severe in hedge fund databases, where reporting is voluntary and failed funds stop reporting, and in equity universes constructed from a current index membership list.

Remedy: use a point-in-time database that records index membership and security existence as of each historical date, including delisted securities and their delisting returns.

Look-ahead bias

Using information that was not available at the time of the decision. Three specific forms:

1. Reporting lag. Using a fiscal-year-end book value on the fiscal year-end date, when it was published three months later. The standard remedy is to lag fundamental data by a conservative period, often four to six months.
2. Restated data. Databases store the restated financials, not the figures as originally reported. A strategy that screens on the restated numbers uses knowledge of the restatement.
3. Index membership. Assuming a security was in the index before its addition was announced.

Data snooping (data mining)

Repeatedly testing variations on the same dataset until one appears significant. With enough attempts, a spurious result is certain: testing 20 independent specifications at a 5% significance level produces roughly one "significant" finding by chance alone.

Its symptoms and remedies:

• Symptom: a strategy with many parameters that were each tuned to the sample; performance that collapses on any small change to the parameters.
• Remedy 1: hold out an out-of-sample period that is never examined until the strategy is final.
• Remedy 2: require a higher significance threshold when many tests have been run, and report the number of specifications tested.
• Remedy 3: demand an economic rationale for the signal before accepting a statistical result. A relationship with no plausible mechanism is more likely to be noise.
• Remedy 4: test on a different market or period entirely.

Related implementation problems that a clean backtest must also address: ignoring transaction costs and market impact, assuming execution at the closing price, ignoring short-sale constraints and borrow costs, and assuming unlimited capacity in illiquid securities.

4. Historical Scenario Analysis

Historical scenario analysis takes the actual returns of a defined past stress episode and applies them to the current portfolio, answering "what would today's positions have done in that event".

Common scenarios: the 1987 crash, the 1998 sovereign and hedge fund crisis, the 2000–2002 technology decline, the 2007–2009 global financial crisis, the 2013 taper episode, the 2020 pandemic shock, and the 2022 rate shock.

The specific insight this exercise supplies, and that VaR does not, is correlation behaviour under stress. A portfolio that looks diversified on average correlations may show that in 2008 every risky asset moved together, so the diversification the risk model reported was not available when it was needed.

---

5. Monte Carlo Simulation

Monte Carlo simulation generates many possible paths by drawing from assumed distributions for the underlying risk factors, then computes the outcome for each path and analyses the resulting distribution.

Procedure:

1. Specify the risk factors and the functional form linking them to portfolio value.
2. Choose distributions for each factor and specify the correlation structure between them.
3. Draw a large number of random scenarios — commonly tens or hundreds of thousands.
4. Reprice the portfolio in each scenario.
5. Analyse the distribution: mean, percentiles, value at risk, conditional value at risk, and the probability of specified outcomes.

Multivariate distributions matter. Drawing each factor independently understates joint tail risk. The standard approach specifies a covariance matrix, and more sophisticated implementations use copulas to model tail dependence — the empirical fact that extreme moves occur together more often than a multivariate normal assumption implies.

Distributional choice is the dominant assumption. Using a normal distribution understates tail frequency; using a Student's t with low degrees of freedom, or bootstrapping from historical returns, produces fatter tails and materially higher tail risk estimates.

Monte Carlo versus historical simulation

Neither dominates. The disciplined practice is to run both: historical simulation for realism about joint behaviour, Monte Carlo to explore the space of events that has not yet occurred.

---

6. Sensitivity Analysis

Sensitivity analysis varies one input at a time — or a small number jointly — and reports how the conclusion changes. In a backtesting context it answers whether the result is robust or an artefact of a specific parameter choice.

What to vary:

• Strategy parameters: the lookback window, the rebalancing frequency, the number of holdings, the signal threshold.
• Cost assumptions: double the assumed transaction cost and see whether the strategy still works.
• Distributional assumptions: re-run a Monte Carlo with a Student's t instead of a normal distribution and compare the tail estimates.
• Sample period: split the history into sub-periods and test each.

The interpretive rule: a strategy whose performance survives reasonable variation in its parameters is more likely to reflect a genuine effect. A strategy that works at a 60-day lookback but fails at 50 and 70 days has been fitted to the sample, and the narrow peak is the signature of data snooping.

Illustration. A researcher reports a backtested information ratio of 1.4 for a momentum strategy using a 12-month formation period, a one-month holding period, and 10 basis points of assumed transaction cost. Sensitivity analysis shows:

The strategy is robust to the formation window but not to realistic transaction costs or to the exclusion of small, illiquid stocks. The honest conclusion is that the reported information ratio is largely a small-cap liquidity premium net of an implausible cost assumption — which is exactly the conclusion a well-built item set will ask for.

Level II traps in this module

1. Treating a single full-sample regression as a backtest; a backtest requires a rolling window.
2. Using restated financial data, or year-end data on the year-end date, both of which are look-ahead bias.
3. Reporting a Sharpe ratio for a negatively skewed strategy without a drawdown or Sortino measure.
4. Believing Monte Carlo is more reliable than historical simulation because it produces more scenarios; more scenarios drawn from a wrong distribution do not add information.
5. Concluding a strategy is validated because it survived one out-of-sample period after twenty in-sample specifications were tested.