## Main Takeaways
* **Forecaster performance can be improved using statistical methods**: By applying a calibration adjustment trained on a particular forecaster, a statistically significant improvement can be made to their Brier score.
* **Improvement is modest**: Brier scores decreased at max by about 0.02, although for some forecasters, this is not insignificant given many were in the 0.15 to 0.20 range.
* **These methodologies are open source and computationally cheap**: The code to apply these adjustments, for instance, to a bot, is available at [this link](https://github.com/Metaculus/forecasting-tools/tree/f7579d3b9188191e5576a02d53d0896bb948c5ad/forecasting_tools/calibration_adjustment).
* **Further work needs to be done to apply calibration adjustments to continuous domained questions**: The Platt scaling method should be relatively easily adaptable.
* **The methodologies in this article are not exhaustive and may be improved**: More methodologies may also be tried, such as using a simple neural network or isotonic regression.
## Introduction
A perfectly calibrated forecaster assigns probability $p$ to events that resolve positively $p\%$ of the time. In practice, most forecasting agents — human or automated — deviate from this ideal. For example, here is the calibration curve of a top professional forecaster:

And here is the calibration curve of one of the top performing bots:

With both the bot and professional, there are regions where their calibration (seen by the yellow dots) strays from the perfect calibration line. From this, a question can be derived: if a forecaster is regularly under- or over-confident with forecasts submitted at probability $p$, could this be corrected for? This question is even more pertinent with bots than professionals, as humans will adjust their thinking subtly over time, but bots can be frozen in place.
This article benchmarks several strategies to explore the feasibility and benefit of applying simple statistical adjustments to a forecaster's inputs to improve calibration.
## Calibration Adjustment Strategies
There are multiple feasible approaches to this problem, but they all follow the same general procedure:
1. Create a train test partitioning of a forecaster's historical forecasts. Forecasts are sorted chronologically, and the earliest 70% are used for training while the most recent 30% form the held-out test set.
2. Train a model based on the historical forecasts of a given forecaster in the training partition.
3. Use the trained model to adjust the predictions in the test partition.
4. For each forecaster, calculate the Brier score of the pre-adjustment and post-adjustment test partition.
5. Test whether there is a statistically significant improvement in Brier scores after adjustment.
### Strategy 1: Uniform Shift
This is the simplest approach. We compute the mean residual across all training observations and add this to every future prediction. This corrects only global mean bias: if a forecaster systematically underestimates by 3 percentage points, the shift is +0.03. It cannot fix local under- or over-confidence.
For multiple-choice questions, this approach has a structural weakness: after adding a constant offset $\delta$ to each raw probability and renormalizing, the adjusted predictions $p_{A,n} \propto p_n + \delta$ converge toward a uniform distribution as $\delta \to \infty$, thereby reducing the information content of the original forecast. We therefore expect this strategy to perform poorly on multiple-choice questions.
### Strategy 2: Step Shift
Building on the uniform shift, the next natural step is to partition the domain into $N$ equal-width subdomains. This is, after all, how Metaculus plots its calibration curves. For each subdomain $1 \le n \le N$, the same procedure as the uniform shift is applied, but trained only on forecasts made in the range $[\frac{n-1}{N}, \frac{n}{N}]$.
The number of steps $N$ is selected automatically by further splitting the training data into an internal train validation set and evaluating candidate values of $N \in [1, 30]$, selecting the value that yields the lowest Brier score on the internal validation set. Note that $N = 1$ is equivalent to a uniform shift.
### Strategy 3: K-Means Shift
Rather than partitioning into $N$ equal-width subdomains, we instead define $K$ cluster center, and assign each forecast to its nearest center. This is trained using K-means clustering, and can be useful for capturing under- or over-confidence in regions with a high density of predictions (for example, near the extremes of $\approx 0$ and $\approx 1$).
The value of $K$ is selected via the same internal validation procedure as $N$ in the step shift adjustment. Note that $K = 1$ is equivalent to a uniform shift.
### Strategy 4: Decision Tree
This model partitions the $[0, 1]$ probability axis into adaptive-width bins (up to 32 bins, each containing at least 30 training observations). For each bin, it stores the empirical resolution rate. At adjustment time, a raw prediction is routed to its bin and replaced by that bin's empirical resolution rate. This is the most flexible approach — it can correct arbitrary bin-level miscalibration — but it is also susceptible to overfitting, as bin-level resolution rates estimated from finite samples may not generalize to held-out data.
### Strategy 5: Logistic Recalibration (Platt Scaling)
Fits an affine transformation in log-odds space: $adjusted = \sigma\!\left(\frac{\operatorname{logit}(p) - \mathrm{bias}}{\mathrm{confidence}}\right)$. This is equivalent to fitting a logistic regression of outcomes on the log-odds of predictions. The adjustment is smooth and monotonic, and its two parameters have direct interpretations in terms of the two most common forms of miscalibration:
* **bias** corrects systematic under- or over-estimation.
* **confidence** corrects under- or over-confidence (i.e., whether predictions are too extreme or too conservative).
This approach can fail in two cases, both of which result in the affected forecaster being excluded from evaluation for this strategy: (1) the training data contains only one resolution class (all "yes" or all "no"), since logistic regression cannot be fit in that case; and (2) the fitted slope is non-positive, indicating that the forecaster's raw predictions are anti-correlated with outcomes, which would produce a non-monotonic recalibration.
## Measuring Forecast Quality: the Brier Score
The **Brier score** is a metric used in forecasting that is equivalent to the mean squared error (MSE) of the forecasts made. This analysis is restricted to binary and multiple-choice questions. A multiple-choice question with $N$ options can be thought of as $N$ binary questions where $\sum_{n=1}^{N} p_n = 1$.
Throughout this notebook, we report $\Delta = \operatorname{Brier}(\text{raw}) - \operatorname{Brier}(\text{adjusted})$. **Positive deltas indicate that the adjuster improved the forecast.**
**A note on comparing binary and multiple-choice Brier scores.** The raw multiple-choice Brier definition $\sum_i (p_i - y_i)^2$ sums squared errors over $K$ options, so its magnitude scales with $K$ and is not on the same $[0, 1]$ scale as the binary Brier score. To make the two more directly comparable, we report the **per-option mean squared error** for multiple-choice questions, i.e. we divide the standard MC Brier by $K$. Even with this normalization, binary and MC Brier are still not perfectly equivalent metrics — the per-question variance and baseline rates differ between question types — so cross-type comparisons of absolute Brier values should be interpreted with care. Within a single question type, deltas (raw vs. adjusted) and the associated $p$-values remain the primary quantities of interest.
## Dataset Selection
The experiment is conducted on data from Metaculus' 2025 Q1 and Q2 AI Benchmarking Tournament. This dataset is particularly well-suited to this analysis: bots forecast on nearly every available question, providing substantially denser per-forecaster training data than is typical for human forecasters. Bots are also easier to apply an automated adjustment to at prediction time, as the adjustment can be inserted with just a few lines of code.
Forecasting agents with fewer than 500 forecasts are excluded to ensure sufficient data for estimating stable, generalizable calibration patterns. The 500-forecast threshold is a practical heuristic chosen to support reliable estimation across all five strategies.
## Model Training
Forecasts are grouped by forecaster, and within each forecaster by question type (binary vs. multiple-choice). A separate calibration model is trained for each (forecaster, question type, strategy) triple. Training and test sets are constructed using a temporal split — the data is sorted chronologically and the earliest 70% of each forecaster's questions are used for training, with the most recent 30% held out for evaluation. This reflects the realistic deployment scenario in which calibration adjustments are learned from historical behavior and applied to future predictions.
An adjustment trained on the pooled data across all forecasters was also evaluated, but was found to be too heavily influenced by the most prolific forecasters and was abandoned in favor of per-forecaster models.
**A note on training-row decomposition.** Binary questions contribute a single $(p, y)$ training observation each. Multiple-choice questions with $K$ options are decomposed into $K$ observations of the form $(p_i, \mathbb{1}[\text{option}_i = \text{resolution}])$, one per option. Because each model is fit on its own question type (binary models never see MC rows and vice versa), this decomposition does not bias one type relative to the other; it does, however, mean that the effective number of training observations backing an MC adjuster is several times the question count, which is worth keeping in mind when interpreting bin counts and model complexity.
### Selected Bin Counts
Both the K-means and step adjusters auto-tune their bin count via a held-out Brier search inside the training data ($k \in [1, 30]$, $n_{\text{buckets}} \in [1, 30]$). The plots below show how the selected complexity is distributed across forecasters, broken out by question type, indicating how much resolution the data actually supports.
Note that for both K-means and step adjustments, a selected value of $k = 1$ or $N= 1$ is equivalent to a uniform shift. Fortunately, this is an infrequent occurrence.

## Results
The following table shows the mean and median Brier scores for both the raw and adjusted forecasts, along with the $p$-value from a one-sided Wilcoxon signed-rank test under the alternative hypothesis that the raw Brier score exceeds the adjusted Brier score.
| | Adjuster | Question Type | Mean Brier Raw | Mean Brier Adjusted | Mean Delta | P Value | Median Brier Raw | Median Brier Adjusted | Median Delta |
| - | ---------------------- | --------------- | -------------- | ------------------- | ---------- | -------- | ---------------- | --------------------- | ------------ |
| 0 | Uniform Shift | binary | 0.18395 | 0.173510 | 0.010440 | 0.001682 | 0.188625 | 0.170535 | 0.018091 |
| 1 | Uniform Shift | multiple choice | 0.16770 | 0.169538 | -0.001838 | 0.999853 | 0.165218 | 0.165751 | -0.000533 |
| 2 | Step Shift | binary | 0.18395 | 0.173119 | 0.010831 | 0.024139 | 0.188625 | 0.173662 | 0.014963 |
| 3 | Step Shift | multiple choice | 0.16770 | 0.163287 | 0.004413 | 0.006016 | 0.165218 | 0.160578 | 0.004639 |
| 4 | K-Means Shift | binary | 0.18395 | 0.176528 | 0.007422 | 0.106071 | 0.188625 | 0.174399 | 0.014227 |
| 5 | K-Means Shift | multiple choice | 0.16770 | 0.163688 | 0.004012 | 0.007965 | 0.165218 | 0.160966 | 0.004251 |
| 6 | Decision Tree | binary | 0.18395 | 0.177222 | 0.006728 | 0.123104 | 0.188625 | 0.173135 | 0.015490 |
| 7 | Decision Tree | multiple choice | 0.16770 | 0.167296 | 0.000404 | 0.366863 | 0.165218 | 0.164870 | 0.000348 |
| 8 | Logistic Recalibration | binary | 0.18395 | 0.167524 | 0.016426 | 0.000523 | 0.188625 | 0.166202 | 0.022423 |
| 9 | Logistic Recalibration | multiple choice | 0.16770 | 0.162343 | 0.005357 | 0.000053 | 0.165218 | 0.159634 | 0.005583 |
The following tables ranks the strategies by $p$-value for binary and multiple choice questions respectively.
| | Adjuster | P Value | Mean Delta | Median Delta |
| - | ---------------------- | -------- | ---------- | ------------ |
| 8 | Logistic Recalibration | 0.000523 | 0.016426 | 0.022423 |
| 0 | Uniform Shift | 0.001682 | 0.010440 | 0.018091 |
| 2 | Step Shift | 0.024139 | 0.010831 | 0.014963 |
| 4 | K-Means Shift | 0.106071 | 0.007422 | 0.014227 |
| 6 | Decision Tree | 0.123104 | 0.006728 | 0.015490 |
### Brier Score Distributions
The histograms below show the per-forecaster Brier score distribution before (gray) and after adjustment (colored) for each strategy. A leftward shift of the colored histogram relative to the gray histogram indicates improvement (lower Brier score). Rows correspond to question type (binary, multiple choice) and columns to each adjustment strategy.

The following chart is a simple plot that compares the improvement in Brier score against the original Brier score pre-adjustment. The hypothesis that this checks is that, in general, the higher the input brier score, the less a model will improve it. A positive slope would agree with that hypothesis.

At least judging from the binary, it seems that the densest cluster of points is along a positive slope, which hints that the improvement by applying an adjuster may not be as strong, or may even be negative.
### Calibration Curves
The chart below visualizes calibration before and after adjustment. In a calibration curve, the x-axis is the predicted probability and the y-axis is the empirical resolution rate for predictions in that bin. A perfectly calibrated forecaster lies on the diagonal $y = x$: of all forecasts assigned probability 70%, the forecasted event should resolve positively 70% of the time.
Each panel compares the raw (gray) calibration curve against one adjuster (colored). The closer the colored curve lies to the diagonal relative to the gray curve, the more the adjuster improved calibration.
The calibration curves are aggregated across all forecasters. Each bin's plotted resolution rate is the fraction of outcomes that resolved positively among all forecasts falling within that probability range, aggregated across all forecasters, for either the raw or adjusted predictions.

When interpreting these graphs, it is important to compare points on the same vertical line rather than along the axis normal to the perfect calibration line, as may be visually intuitive. It is also important to note that these graphs represent the aggregate across multiple forecasters; individual forecasters' calibration curves will look different.
In general, the auto-calibration has improved the visual fit to the diagonal, which is consistent with the conclusions from the hypothesis tests. However, these graphs should be interpreted with care and with reference to the $p$-values reported above. Because they aggregate across multiple forecasters, there can be a degree of smoothing that makes performance appear better than it is at the individual level, as is the case with the decision tree, which shows a visually plausible curve despite failing to reach statistical significance.
## Conclusions
This article has demonstrated that simple statistical calibration adjustments can produce meaningful, statistically significant improvements to forecaster accuracy as measured by the Brier score. Across five strategies of increasing complexity, logistic recalibration (Platt scaling) emerged as the clear best performer, achieving highly significant improvement for both binary and multiple-choice questions. Crucially, this result held across both question types simultaneously, which none of the other strategies achieved as convincingly. The step shift also achieved significance across both types, which is noteworthy given its simplicity.
The remaining strategies present a more nuanced picture. The uniform shift achieved significant improvement for binary questions, consistent with the theoretical argument that a global mean correction is well-suited to scalar predictions. However, as anticipated, it actively harmed performance on multiple-choice questions, where adding a constant offset and renormalizing collapses the adjusted predictions toward uniformity. The K-means shift showed the inverse pattern - significant for multiple-choice but not binary - suggesting that its data-adaptive cluster placement is better suited to the higher-dimensional structure of multiple-choice probability vectors. The decision tree failed to reach significance in either setting, which is most likely attributable to overfitting: its bin-level resolution rate estimates are sufficiently noisy at the available sample sizes that they do not generalize reliably to held-out data.
The results support practical recommendation: using an automated adjustment strategy (such as logistic recalibration) for automated forecasting pipelines is sensible to do by default, especially for simpler, more poorly calibrated forecasting agents. All the strategies used here are computationally trivial to apply at prediction time, and logistic recalibration specifically achieves significant improvement across both multiple choice and binary question types. That being said, better performing forecasting agents may not benefit as much from an adjustment, and some may be negatively impacted. As bots get better, applying an adjustment may not always result in a better performing output, so it would be wise to test an adjuster on some historical forecasts for any given bot that wishes to apply an adjuster before integration. Nonetheless, the methods here are not exhaustive, which warrants further investigation into methods that may perform more favorably on better performing bots.
A natural direction for future work is extending these calibration adjustment strategies to continuous distribution questions. Metaculus represents such forecasts as probability distributions over a bounded range. Miscalibration in this setting takes a different form than in the binary case: a forecaster's distributions may be systematically too narrow (overconfidence), shifted in one direction (bias), or asymmetrically skewed. The standard diagnostic tool is the **probability integral transform (PIT)**, which evaluates the forecaster's cumulative distribution function at each true outcome.
There are a couple strategies as candidates for recalibrating continuous questions. A location-scale recalibration (fitting a linear correction to the forecaster's median and a multiplicative correction to their distribution's width) is the direct generalization of Platt scaling and would be the natural first approach. Isotonic regression applied to the empirical PIT distribution can learn a nonparametric monotone correction to the quantile function without assuming a particular distribution family, analogous to the step shift or decision tree strategies explored here.