The Probability of Backtest Overfitting (PBO) is the fraction of in-sample/out-of-sample partitions where the in-sample winning strategy performs worse than the median strategy out of sample. A PBO of 0.5 means selection carries no information — the in-sample winner is a coin flip out of sample; above 0.5 it is a worse-than-random out-of-sample bet. LLM-augmented research raises backtest-overfitting risk through three structural channels — training-set leakage, prompt-encoded pattern bias, and unbounded candidate generation — but, as the worked cases below show, PBO cleanly catches only one of them. This article walks through the formulation from Bailey, Borwein, López de Prado, Zhu (2014), shows how to compute the score, demonstrates a worked example with 200 candidate strategies, and lays out the mitigation pattern that holds up under adversarial review. Run your own data through /backtest-overfitting-score/ and combine it with /walk-forward-validator/ before paper trading.

What classical OOS testing gives you, and what it leaves on the table

The standard discipline before LLMs touched a backtest was in-sample fitting, out-of-sample evaluation on a held-out window, and walk-forward to test parameter stability across a rolling fit.

IS/OOS catches the obvious case where parameters were fit to noise. If your strategy returns a Sharpe of 2.4 on 2014 to 2020 and 0.3 on 2021 to 2023, the OOS gap is the diagnosis. The fix is simpler parameters or a different signal. Walk-forward extends this by repeating the IS/OOS cut at every step, so you see whether the OOS gap is a one-off artifact of the chosen split date or a persistent feature of the strategy.

These controls are necessary but not sufficient. The hole they miss is selection bias from the candidate pool. If you tested fifty crossover strategies, fifty mean-reversion strategies, and fifty pairs trades, then kept the one with the best walk-forward Sharpe, that single number is biased upward. The expected maximum of fifty noise draws is positive even when the true edge of every strategy is zero.

Walk-forward on the winning strategy alone tells you nothing about how many siblings it had. PBO closes that gap by checking whether the in-sample winner from a random partition also ranks near the top out of sample, or regresses to the middle of the pack.

The PBO formulation in plain English

The 2014 paper by Bailey, Borwein, López de Prado, and Zhu (The Probability of Backtest Overfitting, Journal of Computational Finance) defines PBO through Combinatorially-Symmetric Cross-Validation (CSCV). The procedure is mechanical.

Start with a returns matrix: rows are time, columns are candidate strategies, T observations, N strategies. Chop the time axis into 2S equal-length blocks (S is typically 8, giving 16 blocks; for ten years of daily data, blocks of roughly 150 trading days each). Enumerate every way to split the 2S blocks into two halves of S blocks each: C(2S, S) partitions, 12,870 for S = 8. One half is in-sample (IS), the other out-of-sample (OOS).

For each partition, compute every strategy's Sharpe ratio on the IS half and OOS half independently. Find the strategy with the highest IS Sharpe; call it n*. Rank n*'s Sharpe within the OOS distribution of all N strategies. If n* ranked in the top half of OOS, the IS winner generalized; if in the bottom half, it failed to generalize.

PBO is the fraction of partitions where the IS winner ranked below the OOS median:

PBO = P(λ_OOS_winner ≤ 0)

where λ is the logit of n*'s fractional rank in the OOS Sharpe distribution. Logit zero corresponds to the median, so the inequality is the formal version of "IS winner ranked below median OOS."

Computing PBO: the matrix you need

In code, the algorithm is roughly forty lines.

from itertools import combinations
import numpy as np

def pbo(returns, s=8, sample=None, seed=42):
    """
    returns: np.array shape (N_strategies, T_observations)
    s: half-block count; total partitions = C(2s, s)
    sample: if int, randomly draw that many partitions instead of enumerating
    """
    n, t = returns.shape
    chunk = t // (2 * s)
    blocks = [returns[:, i * chunk:(i + 1) * chunk] for i in range(2 * s)]

    all_partitions = list(combinations(range(2 * s), s))
    if sample is not None and sample < len(all_partitions):
        rng = np.random.default_rng(seed)
        idx = rng.choice(len(all_partitions), size=sample, replace=False)
        partitions = [all_partitions[i] for i in idx]
    else:
        partitions = all_partitions

    below = 0
    for is_idx in partitions:
        is_set = set(is_idx)
        is_mat = np.concatenate([blocks[i] for i in is_set], axis=1)
        oos_mat = np.concatenate(
            [blocks[i] for i in range(2 * s) if i not in is_set], axis=1
        )

        is_sr = is_mat.mean(axis=1) / is_mat.std(axis=1, ddof=1)
        oos_sr = oos_mat.mean(axis=1) / oos_mat.std(axis=1, ddof=1)

        winner = int(np.argmax(is_sr))
        rank = (oos_sr < oos_sr[winner]).sum() / (n - 1)
        clamped = max(0.001, min(0.999, rank))
        logit = np.log(clamped / (1 - clamped))
        if logit < 0:
            below += 1

    return below / len(partitions)

For S = 8 the full enumeration is 12,870 partitions; on a 200-strategy, 2,500-day matrix that runs in under a minute on modern hardware. For larger N or longer T, sampling 500 partitions gives a PBO estimate accurate to ±4 percent. The browser implementation at /backtest-overfitting-score/ uses this exact sampling rule.

What the 0.5 threshold means, and what it does not mean

A PBO of 0.5 says the in-sample winner of a random partition has a 50/50 chance of ranking below the OOS median: the same odds as picking a strategy at random. If the selection process adds zero information, expect PBO around 0.5.

PBO above 0.5 is worse than random. It indicates negative information transfer from IS to OOS, usually because the IS winner is a strategy whose in-sample fit was driven by features that reversed out of sample. PBO below 0.5 indicates positive information transfer: in-sample winning predicts OOS performance better than chance.

Interpretation buckets:

PBO Reading
< 0.2 In-sample winners generalize well. Selection process adds real information.
0.2 to 0.4 Modest selection edge. Combine with PBO confidence interval before deploying.
0.4 to 0.6 Indistinguishable from random selection. Stop.
> 0.6 Anti-selection. The in-sample winner is systematically a worse OOS bet than the median.

What PBO does not tell you: it does not bound the expected OOS Sharpe of the winner. A PBO of 0.1 with a candidate pool of three trivial strategies tells you the best ranks above OOS median, but the absolute OOS Sharpe could still be 0.2. Pair PBO with the Deflated Sharpe Ratio for the absolute-performance question.

PBO also does not catch survivorship bias in the universe (testing only stocks that survived the period), look-ahead leakage in feature construction, or non-stationary regime shifts. It is one statistical control inside a larger validation flow, not a single-number verdict.

How LLM-generated strategies interact with PBO

LLM-generated and LLM-tuned strategies stress the validation flow in three distinct ways. Only one of the three actually shows up as elevated PBO; the other two are PBO's blind spots, which is exactly why PBO is never a solo verdict.

1. The model has seen futures. Pre-training data for any commercial LLM as of 2026 includes financial commentary, academic backtest papers, retail trading blogs, and earnings transcripts dated through the model's cutoff. Ask a model to propose a momentum strategy on USD/JPY for the 2018 to 2023 window and the model has already read post-hoc analyses describing what worked in that exact window. The strategy is not generated from a hypothesis; it is retrieved from a corpus that includes the answer. This is the most dangerous channel and PBO's worst blind spot: a strategy that encodes what worked across the whole historical window looks strong in both CSCV halves, so it ranks high out of sample and pushes PBO down. The only defence is genuine post-cutoff data the model could not have seen — see Case C below.

2. The prompt encodes pattern-matched bias. Even with a careful prompt, instructions like "find a stable signal" or "minimize drawdown" prime the model toward strategies over-represented in the training corpus, and corpus over-representation correlates with strategies that worked historically. When that bias favours patterns that fit one sub-period and reverse in another, PBO does catch it: the in-sample winner regresses below the out-of-sample median across partitions. This is the failure mode PBO was built for, and the one channel of the three that the score surfaces cleanly.

3. The candidate pool is enormous. With a classical strategy search, the candidate count is bounded by what the researcher manually specifies. With LLM-driven generation, it balloons: ten variants per iteration, ten iterations, three time windows produces 300 candidates from a single prompt. This does not push PBO above 0.5 on its own. Adding candidates dilutes the selection signal and drags PBO toward 0.5, not past it — in eight-seed CSCV runs on the worked-example spec below, growing a 30-candidate pool to 300 with pure-noise additions moved mean PBO from 0.38 to 0.47, never systematically above 0.5. Candidate count bites through a different statistic: the Deflated Sharpe Ratio's multiple-testing haircut, whose expected-max-Sharpe benchmark grows with N. Use PBO for selection consistency and DSR for the multiplicity penalty.

The channels compound, but not all through PBO. Leakage (channel 1) hides from PBO entirely; pattern bias (channel 2) is what PBO actually surfaces; candidate count (channel 3) is DSR's job, not PBO's. Running PBO and DSR and a true post-cutoff out-of-sample window together is what closes the gap — any one alone leaves a blind spot an LLM-augmented stack will find.

A worked example: 200 candidates, 10 blocks, 252 partitions

A synthetic example, run with the pbo() code above enumerating all 252 partitions. N = 200 candidate strategies, T = 2,520 daily observations (ten years), S = 5 (10 blocks of 252 days, C(10, 5) = 252 partitions). Daily returns are drawn N(μ, 0.01); a genuine edge sets μ = 0.01 · Sharpe_ann / √252. Figures below are means across eight fixed seeds (11, 22, …, 88).

Case A — pure noise: every candidate has true Sharpe zero. The IS winner's OOS rank is independent of its IS Sharpe under the null, so PBO centres on 0.5. Across the eight seeds it averages 0.55 with a wide 0.28–0.81 spread — a reminder that at 252 partitions and N = 200 the estimate itself is noisy.

Case B — genuine edge: half the candidates carry a real annualized Sharpe of 0.8, half are noise. The IS winner is usually one of the genuine strategies, its OOS rank stays high, and PBO averages 0.22 (0.10–0.32 across seeds) — cleanly below the null. A weaker edge does not separate: at a real Sharpe of only 0.3 the same setup averages PBO 0.48, indistinguishable from noise, because a 0.3 Sharpe over a 1,260-day half is swamped by estimation error.

Case C — whole-window contamination: 20 of the 200 candidates encode what worked across the entire ten-year window (modelled as a strong ~1.5 Sharpe present in every block); the rest are honest. Because the leaked edge sits in every block, it is strong in both CSCV halves: the contaminated strategies win in-sample and also rank at the top out of sample. PBO does not rise — it collapses toward 0 (mean 0.01 here; exactly 0 when the honest pool has no edge at all). Whole-window leakage makes PBO look reassuringly clean while the strategy is worthless on unseen data. CSCV cannot see it, because both halves live inside the contaminated window. Catching it needs genuine post-cutoff out-of-sample data, not a re-slice of the same history.

The lesson inverts the usual intuition: whole-window leakage does not inflate PBO, it suppresses it. A low PBO is necessary but not sufficient — pair it with DSR and with a true out-of-sample window the model never touched. The contamination does not need to be deliberate, or even known to the researcher.

Mitigations

Sorted by cost.

Pre-registered strategy spec. Write down the strategy structure, parameter ranges, and evaluation window before the LLM sees any data. Have the model generate strategies that fit the spec, evaluate on data the model was not asked to inspect, and reject any candidate that uses features not listed. This is the discipline equivalent of pre-registering an academic experiment, and it is the single highest-impact control. PBO drops by 10 to 20 percentage points in practice when the spec is tight and was written first.

Cap on candidate count. Hard ceiling on strategies tested per research loop. A reasonable default is 30 to 50 candidates per project, generated in two or three rounds of LLM brainstorming, with each round seeing only the previous round's failure analysis (not its successes). The cap forces the researcher to engage with each candidate rather than treating the search as a brute-force scan.

Reduce-to-features rather than reduce-to-strategies. Instead of generating 200 strategies and picking the best, ask the LLM to generate 20 candidate features and combine them into a single ensemble model with regularization. The multiplicity correction now applies to features (a much smaller pool) rather than strategies, and regularization handles within-model selection. This is the pattern that survives adversarial review most consistently.

The three combine. Pre-registered spec narrows the hypothesis space, candidate cap bounds the multiplicity, and feature reduction shifts the search from a high-N strategy pool to a low-N feature pool. Each control independently reduces PBO; together they typically halve it.

The three-gate validation flow

PBO is one of three gates: walk-forward, PBO, paper-trading.

Gate one: walk-forward. Run the strategy through /walk-forward-validator/ on a rolling or anchored schedule. Output is a stitched OOS equity curve and a per-fold parameter table. Reject if OOS Sharpe is below 0.4 of IS Sharpe, or if best parameters flip discontinuously fold-over-fold.

Gate two: PBO. Run the candidate pool (every variant tested, not just the winner) through /backtest-overfitting-score/. Reject if PBO is above 0.4. Be honest about the pool: every variant the LLM proposed counts, including the ones you discarded after a quick eyeball.

Gate three: paper trading and probability calibration. Trade the strategy on paper for at least 60 trading days, sized per the LLM-generated probability distribution. Run the resulting probability/outcome pairs through /calibration-dojo/. Reject if the model's stated probabilities are miscalibrated by more than 10 percentage points; miscalibration is what kills Kelly sizing in production.

A strategy that passes all three is not guaranteed to make money. It is guaranteed to be free of the three best-documented retail failure modes: in-sample overfit, selection bias, and probability miscalibration. That clears the runway for the strategy to fail honestly on regime change or execution cost, not on a statistical artifact you should have caught at validation.

This is the PBO-focused entry in the overfitting-diagnostics series: the combinatorial cross-validation formulation and why LLM-augmented searches inflate it. Read alongside:

Connects to

References

  1. Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2014). "The Probability of Backtest Overfitting." Journal of Computational Finance 20(4), 39 to 70. The CSCV formulation and the original PBO derivation.
  2. Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2014). "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance." Notices of the American Mathematical Society 61(5). The accessible companion paper.
  3. Bailey, D. H., & López de Prado, M. (2014). "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality." Journal of Portfolio Management 40(5), 94 to 107. The complementary absolute-Sharpe test.
  4. López de Prado, M. (2018). Advances in Financial Machine Learning, Wiley. Chapter 11 covers backtest overfitting in the ML context, with explicit notes on why naive CV breaks on time series.

Frequently asked questions

What does a PBO score of 0.5 mean for a trading strategy?
A PBO of 0.5 means the in-sample winning strategy has a 50/50 chance of ranking below the out-of-sample median, the same odds as picking a strategy at random. Above 0.5 is worse than random. The article recommends rejecting any candidate pool with PBO above 0.4.
Does using an LLM to generate strategies always inflate PBO?
Not automatically. LLM strategies affect PBO through three channels: prompt-encoded pattern bias does raise PBO, but whole-window training-data leakage is PBO's worst blind spot (it actually suppresses PBO toward zero because the contaminated edge is strong in both CSCV halves). Candidate count inflation is DSR's job, not PBO's.
Why can a near-zero PBO score still indicate a worthless strategy?
Whole-window contamination produces near-zero PBO because the leaked historical edge is strong in every block, so the contaminated strategy wins both in-sample and out-of-sample halves. In the article's worked Case C, 20 contaminated strategies drove mean PBO to 0.01. Those strategies would still fail on genuinely unseen post-cutoff data.
What is the three-gate validation flow the article recommends?
Gate one is walk-forward validation, rejecting if OOS Sharpe falls below 0.4 of IS Sharpe or if best parameters flip discontinuously fold-to-fold. Gate two is PBO over the full candidate pool, rejecting if PBO exceeds 0.4. Gate three is paper trading for at least 60 days with probability calibration checked against outcome frequencies.