A confidence score that an LLM emitted in February is not the same calibration-quality signal as the same score emitted in May. Three mechanisms drive the drift: provider-side tuning (an unannounced refresh of safety, helpfulness, or sampling defaults shifts the model's confidence-vs-correctness mapping; Chen et al. (2023) documented large behaviour swings on a fixed task between two dated snapshots of the same model name), prompt-environment drift (a stable prompt body running against gradually shifting input distributions starts producing systematically over-confident or under-confident outputs), and selection drift (your live workflow only routes the cohort that survived earlier gates, so the calibration test set is now a biased subsample). The detection layer is one cell in a confusion matrix: bucket every emitted probability into deciles, log the actual outcome, compute observed-vs-predicted per bucket on a 30-day rolling window, alert when any bucket diverges by more than one standard error of the empirical proportion. The re-calibration math is isotonic regression or Platt scaling, applied to the most recent N observations and refit weekly. In the synthetic run below, a workflow whose Brier score drifts from 0.19 to 0.23 as calibration degrades is recoverable by inserting isotonic recalibration between the model output and the sizing engine, which re-maps the inflated probabilities back onto their empirical frequencies. Audit production calibration with the Calibration Dojo, and use the Prompt Regression Tester to verify whether a confidence drift is the model, the prompt, or the environment before applying a fix to the wrong layer.
What "calibration" means here
A confidence score is calibrated when, of all the predictions the system labelled "70% probable," roughly 70% turned out true. Reliability is a per-bucket property, measured across many predictions; a single forecast cannot be calibrated or miscalibrated in isolation. The standard scoring rule is the Brier score (mean squared error between predicted probability and binary outcome), with reliability decomposed via the diagram of predicted-vs-observed frequency.
For LLM outputs the typical confidence emission is one of three shapes: an explicit numeric probability ("78%"), a coarse categorical label ("HIGH conviction" mapped to 0.80), or an ordinal scale (1-5 or 1-10) mapped to a probability table. All three suffer from the same drift modes, but the numeric-emission case is easiest to instrument and the categorical case is where drift hides longest before being noticed.
The benchmark to beat is not zero. A naive baseline of always emitting 0.5 produces a Brier score around 0.25 on a 50-50 binary outcome stream; a useful trading-decision classifier should be at 0.18-0.22, and the difference between 0.18 and 0.22 is the difference between a strategy that compounds and one that does not after costs.
How calibration drifts in production
Mechanism 1: provider-side tuning. Frontier model providers ship continuous updates. Some are version-bumped (Claude 4.6 -> 4.7, GPT-5 -> GPT-5-Pro); some are silent revisions to the same version string (safety tuning, reasoning-trace defaults, safety classifier thresholds). The published version string does not capture all of them. A prompt that exploited a specific behavioural quirk of the older revision now hits a slightly different distribution.
The empirical evidence is real but coarse. Chen, Zaharia, and Zou (2023) held the prompt and task fixed and measured substantial behaviour changes between two dated snapshots of the same GPT model name — accuracy on some tasks moving tens of points over a few months, with no version bump. Confidence calibration rides on the same underlying weights, so a silent refresh can move it too. The cause is rarely declared by the provider; the effect is observable downstream and can wreck a workflow that assumed stable calibration.
Mechanism 2: prompt-environment drift. The prompt body has not changed. The system prompt is the same. The model version string is pinned. But the input distribution (the filings the agent reads, the news flow, the macro regime) has shifted. A research prompt calibrated against a low-volatility regime starts producing over-confident outputs in a high-volatility regime because the model's confidence is anchored to the language of the inputs, and high-volatility news is more confident-sounding language.
This mechanism is the most insidious because it produces a smooth, gradual decline. There is no version event to point at, no diff to inspect. The confidence-vs-correctness curve drifts week by week as the underlying environment evolves. The detection has to be statistical, not behavioural.
Mechanism 3: selection drift. A live workflow funnels candidates through gates: research output, calibration check, sizing test. Only the survivors reach the executed-trade set. When you compute calibration on the executed set, you are computing it on a sub-sample selected for the same characteristics that calibration is supposed to measure. The bucket-wise distribution of executed trades is not the bucket-wise distribution of model emissions; the calibration computed on the former does not generalise to the latter.
The fix is to compute calibration on a held-out random sample of all model emissions, not just the executed cohort. This requires logging predictions that did not turn into trades and the ground-truth outcomes for those predictions (price action over the relevant window even though no position was taken). The discipline cost is small; the analytical correctness gain is large.
A worked drift detection: a synthetic research-agent stream
This is a synthetic illustration, not a production run: a constructed two-period stream that reproduces the drift signature cleanly and satisfies the Brier identity exactly. Each period is 340 binary "up over the next 5 days" predictions. The underlying predictability is held fixed across the two periods; only the forecaster's confidence calibration degrades in period 2. Outcomes are drawn from the true per-item probability. The construction is one NumPy block with a fixed seed (data seed 7, latent spread 1.05, period-2 confidence inflation 2.3×), so every figure below reproduces.
Period 1 (calibrated). Brier 0.188. The reliability diagram is clean: bucket-wise predicted probability matches empirical frequency within one standard error in every decile.
Period 2 (over-confident). Brier 0.232. Every bucket from the middle upward is over-confident; the top bucket, predicted around 0.92, resolves true only about 0.80 of the time. Directional discrimination is intact — the damage is in reliability, not resolution.
The full Murphy decomposition, BS = Reliability − Resolution + Uncertainty:
Brier Reliability Resolution Uncertainty
Period 1: 0.188 0.005 0.065 0.250
Period 2: 0.232 0.046 0.061 0.246
Reliability − Resolution + Uncertainty reproduces the Brier column to rounding (0.190 and 0.231). Uncertainty is the irreducible base-rate variance o(1 − o) ≤ 0.25; the base rate sits near 0.5 both periods. Resolution barely moves; reliability climbs almost ten-fold. The model still picks the right direction at roughly the same rate; it just reports too-high confidence. Sizing engines that scale position size with confidence (Kelly, fractional-Kelly) silently take larger positions in the over-confident period and lose money even when the directional call is good.
This is the textbook signature of calibration drift, and it is the one that an output-only QA process misses entirely because the directional accuracy looks fine.
Detection
The minimum-viable detection layer is one rolling computation per day plus an alert.
Logging contract. For every prediction emitted by the agent, persist (run_id, ticker, asof_ts, predicted_probability, model_version, prompt_sha256, outcome_at_horizon, outcome_resolved_ts). Outcome can be resolved asynchronously when the horizon elapses. The store has to be queryable on a 30-day window plus older for backfill.
Rolling reliability check. Each day, compute predicted-vs-observed frequency in each of 10 deciles for the trailing 30 days. Compare to the corresponding 30-day window from 60 days ago (the baseline). For each decile, alert if (current_observed - current_predicted) - (baseline_observed - baseline_predicted) > 1.96 × SE of the empirical proportion. The 1.96 multiplier is the standard 95% normal-approximation cutoff; tighten to 2.58 (99%) for quieter alerting.
def reliability_drift(current, baseline):
"""
current, baseline: lists of (predicted, observed) tuples
Returns deciles where calibration shifted significantly.
"""
drift = []
for d in range(10):
lo, hi = d / 10, (d + 1) / 10
cur = [(p, o) for p, o in current if lo <= p < hi]
base = [(p, o) for p, o in baseline if lo <= p < hi]
if not cur or not base:
continue
cur_obs = sum(o for _, o in cur) / len(cur)
base_obs = sum(o for _, o in base) / len(base)
cur_pred = sum(p for p, _ in cur) / len(cur)
base_pred = sum(p for p, _ in base) / len(base)
delta = (cur_obs - cur_pred) - (base_obs - base_pred)
se = (cur_obs * (1 - cur_obs) / len(cur)) ** 0.5
if abs(delta) > 1.96 * se:
drift.append((lo, hi, delta, se, len(cur)))
return drift
This gives a per-decile signal of where calibration moved. Three or more deciles drifting in the same direction is structural; one decile drifting in isolation is usually noise from a small bucket.
Audit surface. The Calibration Dojo computes the rolling reliability diagram and flags drifting deciles directly from a logged prediction stream. It does not replace your production logging; it consumes the same store and gives the daily diagnostic without writing one more cron.
Distinguishing the cause. Once drift is detected, the next question is whether the cause is the model, the prompt, or the environment. The Prompt Regression Tester re-runs the production prompt against the production model on a held-out fixture set from before the drift began. If the held-out set still calibrates correctly, the cause is environmental drift, not a silent provider change. If the held-out set also calibrates poorly, the cause is upstream of the prompt: either the model has shifted or the prompt has been edited (verify against the prompt-hash log).
Re-calibration math
Once drift is confirmed and the cause is not "the prompt regressed and should be rolled back," the fix is post-hoc recalibration. Two methods cover almost all cases.
Isotonic regression. Fit a monotone non-decreasing function from raw predicted probabilities to recalibrated probabilities, using recent (predicted, observed) pairs. The fit is non-parametric and robust to the shape of the miscalibration. Pseudocode using the standard isotonic algorithm:
from sklearn.isotonic import IsotonicRegression
import numpy as np
# (predicted, observed) for the recalibration window, sorted by predicted
x_train = np.array([p for p, _ in recent_outcomes])
y_train = np.array([o for _, o in recent_outcomes])
cal = IsotonicRegression(out_of_bounds="clip")
cal.fit(x_train, y_train)
# at inference time
def recalibrate(p_raw):
return float(cal.transform(np.array([p_raw]))[0])
Window size: 200-400 observations is the sweet spot for most workflows. Smaller windows track recent drift faster but are noisier; larger windows are stable but slow to react. Refit weekly on a rolling window.
Platt scaling. Logistic regression on raw scores. Fewer parameters than isotonic, hence less overfitting on small samples, but assumes a sigmoid-shaped miscalibration. Useful when you have under 100 observations in the calibration set; default to isotonic above 200.
Where to apply the fix. The recalibration goes between the model output and the sizing engine, not inside the prompt. Asking the model to "self-calibrate" by injecting calibration history into the prompt is unreliable and expensive; a deterministic post-hoc fit is robust and cheap.
model -> raw_probability -> isotonic_recalibrator -> calibrated_probability -> sizing
The calibrator is a small Python object serialised to the same registry as the prompt, with its own version. A weekly refit produces a new calibrator version with the same audit log discipline as a prompt edit.
Pitfalls
Recalibrating without verifying resolution is intact. If resolution (the model's ability to discriminate) has collapsed alongside reliability, recalibration cannot help. A model that produces near-uniform probabilities on every input is uninformative, and remapping uniform probabilities to a different uniform shape does not add information. Check that high-confidence buckets still have higher empirical frequencies than low-confidence buckets before applying recalibration; if they do not, the issue is upstream.
Treating each prompt independently. Calibration drift is correlated across prompts that share a system prompt or tool block. Refitting each prompt's calibrator in isolation wastes data. Pool predictions across prompts that share the same system surface, fit a per-system-prompt calibrator, and use the per-prompt error only to detect prompt-specific issues.
Survivorship in the recalibration window. The recalibration set has to include all emissions, not just executed trades. If only executed trades reach the calibrator, the same selection drift that motivated the article fixes itself only on the survivor cohort, leaving the rest of the workflow uncalibrated. Log all model emissions and outcomes; compute on the full set.
Cliff edges at the boundary. A workflow that checks "is calibrated probability >= 0.65" gets a discontinuity at 0.65. A small recalibration shift around that threshold can flip a large number of decisions in one direction. Mitigate with a smoothed sizing function (Kelly is naturally smooth) rather than a hard threshold; or apply the calibrator's update with a damping factor (e.g. 70% new + 30% prior calibrator) for the first week after a refit.
Ignoring the regression in the prompt-edit log. If a prompt edit shipped the same week the drift began, the cause is almost certainly the edit. Recalibration covers it temporarily; rolling back the edit fixes it permanently. A drift-detection alert that does not reference the prompt-edit log produces wrong root-cause attributions.
Putting it together
Daily: rolling-30-day reliability check across all production prompts; alert on per-decile drift > 1.96 SE versus the 60-days-ago baseline. The check runs in seconds against an indexed predictions table.
Weekly: refit the per-system-prompt isotonic calibrator on the trailing 200-400 observations. Save the new calibrator under a fresh version with a timestamp; deploy via the same gated promotion path as a prompt edit.
Monthly: review the calibrator's version history for trend. A calibrator that has been monotonically pulling probabilities down for three months is a sign that the underlying prompt is producing systematic over-confidence; the right fix at that point is a prompt revision via the Prompt Regression Tester, not another calibration patch.
The combined discipline is what separates a workflow that quietly bleeds money for a quarter from one that catches the leak in the week it opens. The drift is real, the math is solved, and the tooling exists; the missing piece is usually the logging contract, which is the cheapest part to put in place.
References
- Brier, G. W. (1950). "Verification of forecasts expressed in terms of probability." Monthly Weather Review 78(1), 1 to 3. The original definition of the Brier score and the foundation for reliability decomposition.
- Murphy, A. H. (1973). "A new vector partition of the probability score." Journal of Applied Meteorology 12(4), 595 to 600. The reliability/resolution/uncertainty decomposition used in the worked example above.
- Platt, J. (1999). "Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods." Advances in Large Margin Classifiers, MIT Press. The original logistic-regression-based calibration approach.
- Zadrozny, B., & Elkan, C. (2002). "Transforming classifier scores into accurate multiclass probability estimates." Proceedings of KDD '02. The isotonic-regression calibration method.
- Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). "On calibration of modern neural networks." Proceedings of ICML 2017. The empirical demonstration that modern neural nets are systematically over-confident, which generalises directly to LLM-emitted probabilities.
- Chen, L., Zaharia, M., & Zou, J. (2023). "How Is ChatGPT's Behavior Changing over Time?" arXiv:2307.09009. Measures large behaviour shifts on fixed tasks between dated snapshots of the same model name, motivating the silent-revision detection layer.
Frequently asked questions
- What causes calibration drift in an LLM confidence score?
- Three mechanisms drive it. Provider-side tuning — silent model refreshes shift the confidence-vs-correctness mapping even without a version bump. Prompt-environment drift — a stable prompt running against a shifting input distribution produces systematically over- or under-confident outputs. Selection drift — only predictions that survived earlier gates reach the calibration test set, biasing the sample.
- How do I detect calibration drift in a live LLM workflow?
- Compute predicted-vs-observed frequency in each of 10 probability deciles on a rolling 30-day window. Compare each decile to the same window from 60 days ago. Alert when the shift exceeds 1.96 times the standard error of the empirical proportion. Three or more deciles drifting in the same direction is structural; one drifting in isolation is usually noise.
- Should I use isotonic regression or Platt scaling to recalibrate?
- Use isotonic regression when you have 200-400 observations or more — it fits a monotone non-decreasing function without assuming a particular shape and is robust to the form of miscalibration. Use Platt scaling when you have under 100 observations, as its fewer parameters are less prone to overfitting on small samples.
- Why does calibration drift matter if directional accuracy still looks fine?
- Sizing engines that scale position size with confidence take larger positions in over-confident periods even when the directional call is correct. In the synthetic worked example, reliability climbs almost ten-fold (0.005 to 0.046) while resolution barely moves — the model still picks direction correctly but reports inflated confidence, and that silently loses money.