# **AI-driven AI forecasting literature review**

*This is an AI-driven, human-reviewed piece, designed as a research aid to a larger synthesis post found*[ *<u>here</u>*](https://www.metaculus.com/notebooks/43363/overview-of-ai-forecasting-research/)

## **Intro**

As part of running FutureEval (a AI forecasting benchmark) I've collected \~80 links to academic papers, blogs, and write ups related to relevant publications in the AI forecasting field. These are links unafflicated with Metaculus that I've collected and included on the[ <u>FutureEval resources page</u>](https://www.metaculus.com/notebooks/38928/ai-benchmark-resources/#research-papers). As part of writing a larger[ <u>sythesis</u>](https://www.metaculus.com/notebooks/43363/overview-of-ai-forecasting-research/) of the field I wanted to try to find patterns in the literature and which papers to cite. I thought it would be helpful to publish this as well, despite having notably less review than our other publications. As such I've iterated with the AI to try to correct anything obviously wrong, but have not read through and verified the full text (beyond the main takeaways). I have not read all these papers in depth, and have skimmed many. Please note that this literature review intentionally does not include Metaculus research, and focuses on external sources.

Everything (other than this intro and a few minor edits) has been written by an AI agent (Claude Opus 4.7). I have iterated with the agent to double-check interpretations, flag missed findings, and correct glaring problems. The top takeaways have been iterated with most and are generally reliable. I expect there to be some problems in the piece at large though. I also don't necessarily endorse the methodologyr/results/presented-interpretations of all these papers, but I think there is value in aggregating results across the board regardless.

## **Main takeaways for bot makers and new forecasting orgs**

If you are building a FutureEval bot or starting a forecasting org today, these are the 9 things the captured literature says matter most, with the source that most directly supports each one.

1. **Pick a current frontier reasoning model.** Every captured ranking ([<u>ForecastBench</u>](https://www.forecastbench.org/),[ <u>FRI's Oct 2025 update</u>](https://forecastingresearch.substack.com/p/ai-llm-forecasting-model-forecastbench-benchmark),[ <u>Prophet Arena</u>](https://www.prophetarena.co/),[ <u>FutureX</u>](https://futurex-ai.github.io/),[ <u>Fall MiniBench Retrospective</u>](https://predictably.substack.com/p/fall-ai-forecasting-retrospective),[ <u>AIA Forecaster</u>](https://arxiv.org/abs/2511.07678)) puts model choice above scaffolding choice. GPT-5 class, Claude Opus 4.6, and Gemini 3 Pro are the 3 that show up as current leaders across benchmarks.


2. **Agentic search beats one-shot retrieval.**[ <u>AIA Forecaster</u>](https://arxiv.org/abs/2511.07678) measures agentic search worth 0.009 Brier on ForecastBench vs non-agentic, and without search at all the system's live-eval Brier degrades 3.6x. Jordan Berman's[ <u>"One Shot Is Not Enough"</u>](https://thonks.substack.com/p/one-shot-is-not-enough) shows the same pattern on a working commercial bot: ReAct-style iterative retrieval cut his Supervisor invocation rate from 21% to 6%.


3. **Ensemble a handful of diverse runs.**[ <u>AIA Forecaster</u>](https://arxiv.org/abs/2511.07678) beats single-agent via M-agent supervisor reconciliation across 2 ForecastBench rounds. The[ <u>Mantic / Thinking Machines</u>](https://thinkingmachines.ai/news/training-llms-to-predict-world-events/) analysis finds an optimal 4-model ensemble (Gemini 3 Pro, GPT-5, Grok 4, fine-tuned gpt-oss-120b) where the fine-tuned open-weights member is the second-hardest to remove.[ <u>nostream's "On Writing a Bitter-Pilled Metaculus Bot"</u>](https://nostream.substack.com/p/on-writing-a-bitter-pilled-metaculus) attributes its top open-source bot ranking in Fall 2025 primarily to model ensembling, with Grok added specifically as a low-correlation member, independently corroborating Mantic's decorrelation finding.[ <u>Wisdom of the Silicon Crowd</u>](https://arxiv.org/abs/2402.19379) was the founding result but uses n\=31 and very wide equivalence bounds, so weigh it lightly. Median or trimmed-mean aggregation is the common default.


4. **Calibrate post-hoc with Platt scaling.** The 539 bot paper (Phan et al., "LLMs Are Superhuman Forecasters",[ <u>forecast.safe.ai demo</u>](https://forecast.safe.ai/),[ <u>code</u>](https://github.com/centerforaisafety/forecasting)) gets Brier 0.0999 without Platt scaling and 0.0934 with it, on 177 Metaculus retrodiction questions;[ <u>AIA Forecaster</u>](https://arxiv.org/abs/2511.07678) uses Platt scaling as its final stage. Logit-space extremization factor 1.3 to 1.5 is the value that shows up across multiple captured writeups.


5. **Pastcasting works; the magnitude is unmeasured.** Strict-cutoff pastcasting has trained models that demonstrably hold up live: Lightning Rod's[ <u>Foresight-32B</u>](https://arxiv.org/abs/2601.06336) (trained via Future-as-Label on pastcast outcomes) is currently #1 on ProphetArena Sports;[ <u>OpenForecaster</u>](https://www.alphaxiv.org/abs/2512.25070) trained on 52,000 auto-generated pastcast questions and competes with prior-generation closed-weights models on FutureX;[ <u>Turtel et al.'s 14B DPO fine-tunes</u>](https://arxiv.org/abs/2502.05253) reach GPT-4o parity on a held-out Polymarket set (GPT-4o was already not the frontier at publication). The training signal generalises. What is not measured: whether a 0.05 Brier improvement on a pastcast set translates to 0.05, 0.02, or 0.10 on live deployment. No captured paper runs the matched-delta experiment. The other constraint: porous-cutoff pastcasting still overstates skill ([<u>Pitfalls</u>](https://arxiv.org/abs/2506.00723) and[ <u>Simulated Ignorance Fails</u>](https://arxiv.org/abs/2601.13717) measure a 52% Brier gap when LLMs are asked to "pretend they don't know";[ <u>Daily Oracle</u>](https://arxiv.org/abs/2411.08324) shows 21.5% TF / 11.3% MC degradation over 4 years from training contamination), so a strict cutoff is a precondition for any of this to work. Use pastcasting Brier as evidence of direction; use live-resolution benchmarks to calibrate magnitude.


6. **Combine news APIs with native search.**[ <u>Fall AI Forecasting Retrospective</u>](https://predictably.substack.com/p/fall-ai-forecasting-retrospective) reports that switching news search to gpt-5-mini agentic was the single biggest bot-improvement lever of the season.[ <u>Better AI Fact Finding</u>](https://predictably.substack.com/p/better-ai-fact-finding-through-model) independently ranks native-search GPT-5 and Claude 4.5 above stitched-together search APIs.


7. **Simple scaffolds beat elaborate pipelines.** Mohl's[ <u>Helpful vs Harmful Complexity</u>](https://predictably.substack.com/p/helpful-vs-harmful-complexity-for) benchmarks his own multi-researcher / multi-forecaster pipeline against a single GPT-5.1 end-to-end call and finds the single call wins.[ <u>Forecasting Strategies</u>](https://arxiv.org/abs/2406.04446) shows that breakdown, base-rate, both-sides, and news-API prompts do not beat a plain Tetlock-style base prompt once you remove the low-probability-bias artefact.[ <u>nostream's bot retrospective</u>](https://nostream.substack.com/p/on-writing-a-bitter-pilled-metaculus) puts it bluntly: the biggest gains came from boring engineering (ensembling, continuous-question handling, reliability) rather than clever prompting, and a single units bug cost 80 points on one question while missing 10 questions can cost \~150 points.


8. **Open-weights fine-tunes match prior-generation closed-weights leaders.** Turtel's 14B fine-tunes ([<u>LLMs Can Teach Themselves</u>](https://arxiv.org/abs/2502.05253)) hit GPT-4o parity in Feb 2025 when o1 and Claude 3.5 Sonnet were already ahead. Foresight-32B ([<u>Future-as-Label</u>](https://arxiv.org/abs/2601.06336)) beat Qwen3-235B prompting on Metaculus (0.179 vs 0.211), but Qwen3-235B was never a top-2 closed model. Mantic's gpt-oss-120b fine-tune via[ <u>Tinker</u>](https://thinkingmachines.ai/news/training-llms-to-predict-world-events/) climbed past Gemini 3 Pro on their internal Metaculus score (38.6 to 45.8); even there, the optimal 5-model ensemble keeps Gemini 3 Pro, GPT-5, and Grok 4 alongside the fine-tune for decorrelation. Open-weights release cycles lag behind closed frontier moves, so fine-tuning is most useful for decreasing cost-per-inference and as a source of decorrelated ensemble members; for current-frontier single-model quality, frontier APIs are still ahead.


9. **Ship decision-relevant artifacts; watch for self-fulfilling prophecies.** Mo-El's[ <u>Good Forecasts, Bad Products</u>](https://mo-el.xyz/blog/good-forecasts-bad-products/) makes the field's cleanest product-side point: Aaru and Simile each raised 9-figure rounds selling synthetic-population simulations; more generalized forecasters have not. Decide product shape before optimizing on accuracy. Forethought's[ <u>Design Sketches</u>](https://www.forethought.org/research/design-sketches-tools-for-strategic-awareness) names 3 buildable product shapes (ambient superforecasting, scenario planning on tap, automated OSINT). Also consider prediction externalities. Demski's[ <u>Parable of Predict-O-Matic</u>](https://www.lesswrong.com/posts/SwcyMEgLyd4C3Dern/the-parable-of-predict-o-matic) brings up the concern of self-fulfilling prophecies.



## **Scope and methodology**

* **Input set: 82 indexed non-Metaculus sources.** Every external URL on the FutureEval bot resources page as of 2026-05-02 (after dropping one low-signal paper link), plus 4 posts added during human iteration. Index breakdown: 25 research papers, 20 community blog posts, 10 benchmarks, 27 companies / projects.
* **Article scope: papers and blogs.** The synthesis below covers the 25 papers and 20 blogs. The 10 benchmarks and 27 companies / projects were captured as supporting context, are referenced inline where they appear in the captured papers and blogs, but do not get their own sections in this version of the article.
* **Fetch protocol: 3+ attempts per source.** Order was (1) direct WebFetch on the canonical URL, (2) browser-tool navigation + snapshot, (3) web search for a mirror, summary, or archived copy. For arXiv specifically, the /html/\ form and the alphaxiv mirror were also tried before giving up.
* **Failed-fetch checkpoint: human filled the gaps.** The human reviewer supplied 3 paper PDFs (Consistency Checks 2412.18544, LLMs Can Teach Themselves 2502.05253, LLMs Are Superhuman Forecasters / 539 Phan), and markdown captures for 4 blog posts that did not fetch cleanly.
* **Capture rate: 58 of 82 sources citable at the experimental-number level.** Final aggregate across all 82 indexed sources: 51 fully captured on crawl, 23 snippet or landing-page only (mostly the company / project entries that ship marketing copy rather than a public technical artefact), 1 failed fetch (Fate Systems), 7 user-supplied after the checkpoint (3 PDFs + 4 blog captures).
* **Human iteration: review, challenge, enumerate.** After the first draft, a human reviewer iterated with the agent to double-check interpretations that looked off (for example, whether the MCQ status-quo-prior claim had enough evidence to be a main takeaway, and whether the bot descriptions on the Metaculus resources page were in scope), and to enumerate findings that had been missed (notably the positive-pastcasting cases in takeaway 5 and the Evaluation methodology section).
* **Working artefacts.** Per-source notes live in the authors' working folder (other/literature\_review\_v3\_inputs/) under papers/, blogs/, benchmarks/, and companies/ (the latter 2 supporting the inline references in this article), plus a sources\_index.md, a quantitative\_theme\_review.md with category counts and cross-references, and a methodology\_log.md with the full fetch-attempt history.

## **Topical paper and blog groupings**

Every paper and blog post that cleared the snippet bar is placed in one or more of these 8 categories. Each category lists its sources up top and is followed by 3 to 5 bullets capturing the consensus and the genuine disagreements inside it. Benchmarks and companies are referenced inline where the captured papers and blogs cite them.

### **Human-vs-LLM baselines**

Papers and blogs:[ <u>ForecastBench 2409.19839</u>](https://arxiv.org/abs/2409.19839),[ <u>Schoenegger & Park 2310.13014</u>](https://arxiv.org/abs/2310.13014),[ <u>Silicon Crowd 2402.19379</u>](https://arxiv.org/abs/2402.19379),[ <u>Approaching Human-Level 2402.18563</u>](https://arxiv.org/abs/2402.18563), 539 / Phan,[ <u>Daily Oracle 2411.08324</u>](https://arxiv.org/abs/2411.08324),[ <u>Reasoning and Tools 2408.12036</u>](https://arxiv.org/abs/2408.12036),[ <u>BrainBench Nature 2024</u>](https://www.nature.com/articles/s41562-024-02046-9),[ <u>AI-Augmented Predictions MIT</u>](https://arxiv.org/abs/2402.07862),[ <u>AIA Forecaster 2511.07678</u>](https://arxiv.org/abs/2511.07678),[ <u>Contra Papers</u>](https://www.lesswrong.com/posts/uGkRcHqatmPkvpGLq/contra-papers-claiming-superhuman-ai-forecasting),[ <u>FRI ForecastBench update</u>](https://forecastingresearch.substack.com/p/ai-llm-forecasting-model-forecastbench-benchmark),[ <u>Training LLMs with Mantic</u>](https://thinkingmachines.ai/news/training-llms-to-predict-world-events/).

* **Consensus: LLMs hold their own against average forecasters, still trail the top tier.** Schoenegger & Park's Q3 2023 Metaculus Cup result was GPT-4 Brier 0.20 vs human crowd 0.07 (n\=23 questions, Cohen's d\=0.94) and not statistically distinguishable from a 50%-on-everything baseline. By Oct 2025, FRI's Brier-Index leaderboard puts GPT-4.5 at 0.101, the median public forecaster at 0.22 (ranked #22, down from #2 a year earlier), and superforecasters at 0.081. The gap to superforecasters is 0.02; the gap to average public forecasters has flipped. The strongest captured parity claim is[ <u>AIA Forecaster</u>](https://arxiv.org/abs/2511.07678) at Brier 0.108 vs superforecaster median 0.111 across 2 ForecastBench rounds, but the paper itself classifies this as a "retrospective evaluation" (backtest) on already-resolved questions, and bounds rather than precludes foreknowledge leakage via a paper-internal 1.65% audit (not an independent replication). AIA's only prospective / live forward-looking experiment is n\=64 closed markets over Aug 15 to Aug 26 2025 against **market consensus, not superforecasters**, with the authors noting "with only 64 markets we are unable to statistically distinguish these two results." No paper in the captured literature has a prospective (live forward-looking) superforecaster-parity result against a live superforecaster panel.
* **Consensus: domain-specific LLMs meaningfully exceed domain experts.** BrainBench has GPT-4 at 81.4% vs human neuroscience experts 63.4% on forced-choice abstract completion, a large effect; BrainGPT (LoRA on neuroscience corpus) adds 3% on top. The MIT AI-Augmented Predictions RCT (N\=991) shows humans paired with a GPT-4-Turbo assistant improve 24-28%, which is a consistent direction of effect at a much larger sample size than the Metaculus-style studies.
* **Disagreement: every published "parity" or "superhuman" claim against superforecasters is a backtest, not a prospective result.** Silicon Crowd's "statistically equivalent to human crowd" uses n\=31 Metaculus questions and a Δ Brier ≤ 0.081 equivalence bound so wide that the[ <u>Contra Papers critique</u>](https://www.lesswrong.com/posts/uGkRcHqatmPkvpGLq/contra-papers-claiming-superhuman-ai-forecasting) points out "always predict 50%" would also qualify; the raw numbers are LLM-crowd Brier 0.20 vs human-crowd 0.19, with the LLM ensemble slightly losing. Halawi et al.'s "approaching human-level" Brier 0.179 vs crowd 0.149 is a 0.03 gap, above the paper's own "large margin" threshold of 0.02, so the crowd wins clearly in expectation.[ <u>Reasoning and Tools</u>](https://arxiv.org/abs/2408.12036)'s 0.169 vs 0.172 on n\=201 Manifold questions is a 0.003 delta on a play-money baseline that Contra Papers flags for temporal-leakage methodology. The Phan 539 paper's 0.0934 vs 0.0952 Platt-scaled on n\=177 retrodiction questions does not replicate on other sets and has calibration leakage. AIA Forecaster is the most methodologically careful of these (multi-round backtest, foreknowledge audit, transparent failure on MarketLiquid where it underperforms market consensus 0.126 vs 0.111) but it is structurally the same kind of claim Contra Papers warns about: a retrospective evaluation whose integrity depends on how well "as-of-date" hygiene was enforced on a model that may have seen related news during pretraining. The live forward-looking complement is n\=64 against markets, not superforecasters. Until a prospective head-to-head against a co-submitting superforecaster panel exists, treat parity claims as backtested-and-audited, not field-validated.

### **Scaffolding, ensembling, and agentic architectures**

Papers:[ <u>Silicon Crowd 2402.19379</u>](https://arxiv.org/abs/2402.19379),[ <u>Approaching Human-Level 2402.18563</u>](https://arxiv.org/abs/2402.18563),[ <u>Reasoning and Tools 2408.12036</u>](https://arxiv.org/abs/2408.12036),[ <u>Tell Me Why 2502.13410</u>](https://arxiv.org/abs/2502.13410),[ <u>Informed Forecasting 2505.10213</u>](https://arxiv.org/abs/2505.10213),[ <u>PandemicLLM Nature 2025</u>](https://www.nature.com/articles/s43588-025-00798-6),[ <u>Donald Trumps in the Virtual Polls 2411.01582</u>](https://arxiv.org/abs/2411.01582),[ <u>AIA Forecaster 2511.07678</u>](https://arxiv.org/abs/2511.07678),[ <u>Future-as-Label 2601.06336</u>](https://arxiv.org/abs/2601.06336),[ <u>More Isn't Always Better 2603.22152</u>](https://arxiv.org/abs/2603.22152). Blogs:[ <u>Helpful vs Harmful Complexity</u>](https://predictably.substack.com/p/helpful-vs-harmful-complexity-for),[ <u>Better AI Fact Finding</u>](https://predictably.substack.com/p/better-ai-fact-finding-through-model),[ <u>Berman "I'm a Superforecaster"</u>](https://thonks.substack.com/p/im-a-superforecaster-heres-what-happened),[ <u>Mantic / Thinking Machines</u>](https://thinkingmachines.ai/news/training-llms-to-predict-world-events/),[ <u>Comatose Savant</u>](https://thonks.substack.com/p/the-comatose-savant),[ <u>One Shot Is Not Enough</u>](https://thonks.substack.com/p/one-shot-is-not-enough),[ <u>nostream "On Writing a Bitter-Pilled Metaculus Bot"</u>](https://nostream.substack.com/p/on-writing-a-bitter-pilled-metaculus).

* **Consensus: a handful of diverse runs beats one.** AIA Forecaster's M-agent supervisor reconciliation beats single-agent across 2 ForecastBench rounds. The Mantic / Thinking Machines analysis identifies an optimal 4-model ensemble in which a fine-tuned open-weights member is the second-hardest to remove, showing the gain comes from error decorrelation rather than raw ensemble size. nostream's bot retrospective independently credits ensembling as the single largest contributor to Q3 2025 performance and added Grok specifically as a low-correlation member. Halawi's trimmed-mean aggregation across multiple LM scratchpads is baked into their pipeline. Beyond \~5 runs the captured literature does not show clear gains (More Isn't Always Better reports that pattern for human-advisor panels, which is suggestive but not direct evidence for LLM ensembling). The caveat: Silicon Crowd, often cited as the crowd-matching ensemble result, uses n\=31 and bounds too wide to support that reading.
* **Consensus: agentic search beats one-shot search.** AIA Forecaster's search ablation on ForecastBench is clean: search-A (QA API) > search-B (general web) > non-agentic > no search (Brier 0.114 → 0.117 → 0.123), and removing search in a live forward-looking eval degrades Brier 3.6x. Berman's One Shot Is Not Enough independently reports a 21% to 6% drop in Supervisor invocations after switching to agentic retrieval. Reasoning and Tools reports a similar qualitative pattern on n\=201 Manifold questions, but Contra Papers notes the Manifold baseline is play-money and temporally leak-prone, so the 0.169 vs 0.172 delta should not be weighted heavily on its own.
* **Disagreement: multi-agent value beyond ensembling is split.** AIA Forecaster's supervisor + Platt scaling (rigorous, multi-round), Approaching Human-Level's retrieval-plus-SSFT pipeline (small margin, humans still ahead), and Tell Me Why's rationale-deliberation mechanism (theoretical) argue that structured multi-agent deliberation adds value. Mohl's Helpful vs Harmful Complexity and Forecasting Strategies both argue the opposite at current frontier-model quality: a single GPT-5.1 end-to-end call is competitive with a Delphi pipeline at a fraction of the cost. The resolution may be that search + calibration carry most of the weight; deliberation adds less than advertised.

### **Calibration, hallucination, and overconfidence**

Papers:[ <u>Tell Me Why 2502.13410</u>](https://arxiv.org/abs/2502.13410),[ <u>Forecasting Strategies 2406.04446</u>](https://arxiv.org/abs/2406.04446), 539 / Phan,[ <u>More Isn't Always Better 2603.22152</u>](https://arxiv.org/abs/2603.22152),[ <u>Consistency Checks 2412.18544</u>](https://arxiv.org/abs/2412.18544). Blogs:[ <u>Minibench Analysis 1</u>](https://predictably.substack.com/p/minibench-analysis-1),[ <u>Check Your Priors</u>](https://predictably.substack.com/p/check-your-priors),[ <u>Please Don't Make Things Up</u>](https://predictably.substack.com/p/user-says-please-dont-make-things),[ <u>Comatose Savant</u>](https://thonks.substack.com/p/the-comatose-savant).

* **Consensus: LLM forecasters carry systematic priors.** Forecasting Strategies measures a negativity bias on 341 binary Google-internal market events where Basic-prompt P(event) + Reversed P(event) sums to about 0.65 with PaLM 2, so the whole apparent benefit of superforecasting-style prompts turns out to be a low-probability bias under weighted Brier. Consistent with this, Mohl's Check Your Priors observes "doesn't change" capture 60% probability on 10 of 13 Google Trends 3-way MCQ questions in 1 MiniBench round (small N, observational). The broader pattern is well-supported; the MCQ-specific status-quo collapse is a plausible-but-thinly-evidenced special case.
* **Consensus: post-hoc calibration works cheaply.** Phan's 539 paper drops Brier from 0.0999 to 0.0934 with Platt scaling on 177 Metaculus questions; AIA Forecaster adopts Platt scaling as its final stage. Logit-space extremization factor 1.3 to 1.5 is the value reported across multiple captured writeups.
* **Disagreement: hallucination remediation approach is unsettled.** Mohl's Please Don't Make Things Up ranks model choice + RAG + ReAct as the highest-ROI levers; Berman's Comatose Savant favours structured context-setting ("you are a post-cutoff coma patient") and named-uncertainty prompts. The two approaches are complementary, but on current benchmarks neither has been shown to dominate.

### **Evaluation methodology and contamination**

Papers:[ <u>ForecastBench 2409.19839</u>](https://arxiv.org/abs/2409.19839),[ <u>Consistency Checks 2412.18544</u>](https://arxiv.org/abs/2412.18544),[ <u>Context is Key 2410.18959</u>](https://arxiv.org/abs/2410.18959),[ <u>Daily Oracle 2411.08324</u>](https://arxiv.org/abs/2411.08324),[ <u>Pitfalls 2506.00723</u>](https://arxiv.org/abs/2506.00723),[ <u>Simulated Ignorance Fails 2601.13717</u>](https://arxiv.org/abs/2601.13717),[ <u>Question Generation 2601.22444</u>](https://arxiv.org/abs/2601.22444),[ <u>Scaling Open-Ended Reasoning 2512.25070</u>](https://www.alphaxiv.org/abs/2512.25070). Blogs:[ <u>Contra Papers</u>](https://www.lesswrong.com/posts/uGkRcHqatmPkvpGLq/contra-papers-claiming-superhuman-ai-forecasting),[ <u>Talkie vintage LM</u>](https://talkie-lm.com/introducing-talkie).

* **Consensus: porous-cutoff pastcasting overstates skill.** Simulated Ignorance Fails measures a 52% Brier gap between "pretend you don't know" and true pre-cutoff ignorance across 9 models; Daily Oracle shows LLM prescience degrades 21.5% TF / 11.3% MC over 2020 to 2024 from training contamination. The Phan 539 retrodiction set on n\=177 Metaculus questions is the contested canonical case.
* **Consensus: strict-cutoff pastcasting produces credible-looking numbers.** Halawi et al. pastcasted on 914 Metaculus-style questions opened after their cutoff and got Brier 0.179 (crowd 0.149); Turtel et al.'s DPO fine-tune was evaluated on 2,300 Polymarket questions resolved Dec 25 2024 to Jan 23 2025 (after the Phi-4 and DeepSeek-R1 cutoffs) at Brier 0.197 (matches GPT-4o); Talkie's 1931-cutoff vintage LM is the cleanest form: a model that literally cannot know anything past 1931 can be pastcast on any later event without contamination. The working pattern is a temporally held-out test set, a search-date filter, and large enough N that small-N noise doesn't dominate.
* **pastcasting has a positive effect; magnitude transfer is unmeasured.** Pastcast-trained models reach competitive performance on live benchmarks, generally at the prior-generation closed-weights tier. Lightning Rod's[ <u>Foresight-32B</u>](https://arxiv.org/abs/2601.06336) (trained via Future-as-Label) is currently #1 on ProphetArena Sports and top-5 on ForecastBench. OpenForecaster trained on 52,000 pastcast questions and competes on FutureX below the current leaders. Turtel et al.'s 14B DPO fine-tunes reach GPT-4o parity on a held-out Polymarket set, though GPT-4o was no longer the frontier at publication. Future-as-Label also reports cross-distribution transfer: 0.243 → 0.198 (Δ 0.045) on in-distribution synthetic, 0.247 → 0.179 (Δ 0.068) on out-of-distribution Metaculus, so the gain on the harder transfer set is actually larger than in-distribution. AIA Forecaster (not finetuned) reports backtested ForecastBench Brier 0.108 and a separate n\=64 live forward-looking Brier of 0.1002 on closed markets against market consensus, hinting that scaffolding-and-calibration gains may transfer but underpowered to confirm it. What is missing: a same-method side-by-side of pastcast-Brier delta and live-Brier delta on the same model, in the same units, at a sample size where small-N noise does not dominate. Until that is published, pastcasting Brier should be read as evidence of direction, not as a calibrated predictor of live-deployment magnitude.
* **Disagreement: consistency as Brier substitute is open.** Consistency Checks shows the Cond (conditional-probability) check correlates R^2 > 0.9 with Brier; but their own ArbitrageForecaster ablation shows that training for consistency overfits and does not transfer to held-out checks or to Brier. The open question is whether a different training signal could inherit the evaluation benefit without the overfitting.

### **Reinforcement learning and finetuning**

Papers:[ <u>Approaching Human-Level SSFT 2402.18563</u>](https://arxiv.org/abs/2402.18563),[ <u>LLMs Can Teach Themselves DPO 2502.05253</u>](https://arxiv.org/abs/2502.05253),[ <u>Outcome-based RL 2505.17989</u>](https://arxiv.org/abs/2505.17989),[ <u>Future-as-Label 2601.06336</u>](https://arxiv.org/abs/2601.06336),[ <u>Scaling Open-Ended Reasoning 2512.25070</u>](https://www.alphaxiv.org/abs/2512.25070),[ <u>BrainBench / BrainGPT Nature 2024</u>](https://www.nature.com/articles/s41562-024-02046-9),[ <u>AIA Forecaster 2511.07678</u>](https://arxiv.org/abs/2511.07678) (no-finetuning baseline). Blog:[ <u>Training LLMs to Predict World Events (Mantic)</u>](https://thinkingmachines.ai/news/training-llms-to-predict-world-events/).

* **Consensus: outcome RL lifts open-weights small models to prior-generation closed-weights parity.** Turtel et al.'s Phi-4 14B and DeepSeek-R1 14B fine-tunes tied GPT-4o (Brier 0.197 vs 0.196), but o1 and Claude 3.5 Sonnet were already ahead by Feb 2025. Foresight Learning trains Qwen3-32B to beat Qwen3-235B prompting; Qwen3-235B was never a top-2 closed model. OpenForecaster trains an 8B Qwen3 to compete with prior-generation 100B+ closed-weights peers on FutureX, below the current FutureX leaders. In every comparison, the closed baseline is a strong-but-not-current model.
* **Consensus: self-play beats human-curated rationales.** All of Turtel's DPO, Turtel's ReMax-based outcome RL, and Max Planck's GRPO stack are fully autonomous training loops that only need a resolved-outcome stream. Ranking-by-outcome is a cleaner training signal than human-written reasoning chains.
* **Disagreement: finetuning value vs current frontier is split.** Mantic's Tinker RL on gpt-oss-120b moved 38.6 to 45.8 baseline points and crossed past Gemini 3 Pro on Mantic's internal Metaculus test, but Mantic's deployed 5-model ensemble still keeps Gemini 3 Pro, GPT-5, and Grok 4 alongside the fine-tune for decorrelation rather than replacement. AIA Forecaster ties superforecasters with no finetuning at all on a retrospective / backtested ForecastBench evaluation (parity not yet replicated prospectively). The operating consensus: finetune if you have open weights and an RL budget for a cost-per-inference and ensembling edge; use prompt + agentic scaffolding if you are a pure frontier API customer.

### **Decomposition and structured reasoning**

Papers:[ <u>Forecasting Strategies 2406.04446</u>](https://arxiv.org/abs/2406.04446),[ <u>Tell Me Why 2502.13410</u>](https://arxiv.org/abs/2502.13410),[ <u>AIA Forecaster 2511.07678</u>](https://arxiv.org/abs/2511.07678). Blogs:[ <u>Recursive Forecasting (AlignmentForum)</u>](https://www.alignmentforum.org/posts/q2zYtNsh62SCphitt/recursive-forecasting-eliciting-long-term-forecasts-from).

* **Consensus: explicit breakdown prompts do not help.** Forecasting Strategies tests base-rate, breakdown, both-sides, crowd, and news-API prompts against a plain Tetlock-style base and finds no wins once weighted Brier is used; the gains are a low-probability bias artefact.
* **Consensus: decomposition helps with clean base rates.** The AIA Forecaster supervisor reconciles sub-forecasts where each is separately estimable, and Recursive Forecasting proposes chaining short-horizon predictions of the next-step prediction (TD-learning flavour) for long-horizon questions.
* **Disagreement: role of explicit rationales is unsettled.** Tell Me Why frames rationales as atomic-signal decompositions of belief and shows a deliberation mechanism where truthful report is a perfect Bayesian equilibrium; the empirical Forecasting Strategies work says rationale-based decomposition does not translate to Brier improvement on current models. Unresolved.

### **Domain-specific applications**

Papers:[ <u>Context is Key 2410.18959</u>](https://arxiv.org/abs/2410.18959) (time series + text),[ <u>Informed Forecasting 2505.10213</u>](https://arxiv.org/abs/2505.10213) (time series + covariates),[ <u>PandemicLLM Nature 2025</u>](https://www.nature.com/articles/s43588-025-00798-6) (health),[ <u>Donald Trumps in the Virtual Polls 2411.01582</u>](https://arxiv.org/abs/2411.01582) (politics),[ <u>BrainBench / BrainGPT Nature 2024</u>](https://www.nature.com/articles/s41562-024-02046-9) (science). Benchmark:[ <u>LexArena</u>](https://lexarena.ai/) (legal / regulatory).

* **Consensus: domain context is the time-series lever.** Context is Key shows a direct-prompt LLM beats foundation time-series models when essential text context is supplied; Informed Forecasting cuts RMSE 47% to 76% on flu, call-centre, and airline data just by adding coupled key-value prompts with a well-chosen calendar covariate.
* **Consensus: multimodal reasoning helps health forecasting.** PandemicLLM reframes COVID hospitalisation-trend forecasting as multimodal text reasoning over policy, genomics, demographics, and time series across 50 US states, and outperforms domain-specific baselines.
* **Disagreement: domain finetuning vs retrieval comparison is open.** BrainBench shows BrainGPT (LoRA on a 20-year neuroscience corpus on LLaMA-2-7B) beats the base model by 3% but the non-finetuned GPT-4 already beats human experts 81.4% to 63.4% on the task; the Informed Forecasting results argue retrieval-of-covariates is the bigger lever. No clean head-to-head exists in the captured literature.

### **Markets, products, and real-world deployment**

Papers:[ <u>AIA Forecaster 2511.07678</u>](https://arxiv.org/abs/2511.07678),[ <u>Outcome-based RL 2505.17989</u>](https://arxiv.org/abs/2505.17989),[ <u>AI-Augmented Predictions (MIT)</u>](https://arxiv.org/abs/2402.07862). Blogs:[ <u>Forethought Design Sketches</u>](https://www.forethought.org/research/design-sketches-tools-for-strategic-awareness),[ <u>Berman "I'm a Superforecaster"</u>](https://thonks.substack.com/p/im-a-superforecaster-heres-what-happened),[ <u>Fall AI Forecasting Retrospective</u>](https://predictably.substack.com/p/fall-ai-forecasting-retrospective),[ <u>Good Forecasts Bad Products</u>](https://mo-el.xyz/blog/good-forecasts-bad-products/),[ <u>The Death and Life of Prediction Markets at Google</u>](https://asteriskmag.com/issues/08/the-death-and-life-of-prediction-markets-at-google),[ <u>The Parable of Predict-O-Matic</u>](https://www.lesswrong.com/posts/SwcyMEgLyd4C3Dern/the-parable-of-predict-o-matic),[ <u>Real-Life Examples of Prediction Systems Interfering</u>](https://www.lesswrong.com/posts/6bSjRezJDxR2omHKE/real-life-examples-of-prediction-systems-interfering-with).

* **Consensus: accuracy alone does not fund companies.** Mo-El's Good Forecasts, Bad Products compares funded simulation companies (Aaru, Simile) with equally accurate forecasters and documents the valuation gap. Forethought's Design Sketches names 3 buildable product shapes (ambient superforecasting, scenario planning on tap, automated OSINT), all of which ship decision-relevant artefacts rather than probability points.
* **Consensus: AI adds signal in liquid markets.** AIA Forecaster matches superforecasters on a backtested ForecastBench evaluation and adds signal as an ensemble member to liquid-market consensus on its retrospective MarketLiquid benchmark (AIA+market 0.106 beats either alone); its prospective complement is a small n\=64 live forward-looking eval against market consensus only. Turtel's Outcome-based RL paper shows a 14B model earns +10% ROI in a simulated Polymarket loop against market consensus.
* **Consensus: AI assistants lift human forecaster accuracy.** Schoenegger et al.'s MIT RCT (N\=991) reports 24% to 28% accuracy improvement when humans were given a GPT-4-Turbo assistant, including a "superforecasting" prompt and a deliberately noisy control assistant. The effect is real at the consumer assistant level as well as at the expert level.
* **Open question: second-order effects of deployed forecasters.** Forecasts that shape decisions also shape outcomes. Demski's[ <u>Parable of Predict-O-Matic</u>](https://www.lesswrong.com/posts/SwcyMEgLyd4C3Dern/the-parable-of-predict-o-matic) (LessWrong 2019) makes the theoretical case: a system that optimises prediction error can drift toward self-fulfilling outputs, because predictable futures are easier to predict (the parable's "boot stamping on a face forever is reliably predictable" intuition). Sempere's follow-up[ <u>Real-Life Examples of Prediction Systems Interfering with the Real World</u>](https://www.lesswrong.com/posts/6bSjRezJDxR2omHKE/real-life-examples-of-prediction-systems-interfering-with) (LessWrong 2020) catalogues a multitude of empirical instances of predictions (usually negatively) affecting the world. Schwarz's Gleangen retrospective adds a real-world fragment from a working corporate market: a Google senior executive intervened on an underperforming hire after Prophit's market gave the deadline a low probability. None of the captured technical papers builds explicit conditional forecasting (predict the outcome given that I forecast X vs Y, then choose how to report) or measures self-fulfilling attractors in long-running deployment.
* **Consensus: operationalising forecasts inside companies is the hard part.** Dan Schwarz's[ <u>"The Death and Life of Prediction Markets at Google"</u>](https://asteriskmag.com/issues/08/the-death-and-life-of-prediction-markets-at-google) is a first-person retrospective on running both Prophit (Google 2005-2011, \~20% of employees bet) and Gleangen (Google 2020-2022, 15,000 bettors, over 1,000 monthly traders, exceeding Metaculus or Manifold at the time). Two lessons align with Good Forecasts, Bad Products: (a) forecasting competitors is easier and more valuable than forecasting internal progress — Gleangen asked "Will Google integrate LLMs into Gmail?" when executives actually wanted "Will Microsoft integrate LLMs into Outlook?", (b) corporate prediction markets must be sold to managers like enterprise software, not evangelised like consumer growth. Anthropic's 2024 Manifest announcement of an internal market with "focus on decision-makers" is an explicit application of this lesson. The Gleangen dataset underlies Pratt et al.'s[ <u>Forecasting Strategies</u>](https://arxiv.org/abs/2406.04446) paper, so the corporate-market data directly feeds the academic LLM-forecasting literature.

## **Finetuning**

Finetuning for forecasting has gone from a niche research trick in 2024 to a practical edge for small-team open-weights bots in 2026. The deeper question is whether it currently matches frontier APIs, whether self-play on resolved outcomes is enough to train it, and where it fits in a deployed stack.

Papers:[ <u>Approaching Human-Level SSFT 2402.18563</u>](https://arxiv.org/abs/2402.18563),[ <u>LLMs Can Teach Themselves DPO 2502.05253</u>](https://arxiv.org/abs/2502.05253),[ <u>Outcome-based RL 2505.17989</u>](https://arxiv.org/abs/2505.17989),[ <u>Future-as-Label 2601.06336</u>](https://arxiv.org/abs/2601.06336),[ <u>Scaling Open-Ended Reasoning / OpenForecaster 2512.25070</u>](https://www.alphaxiv.org/abs/2512.25070),[ <u>BrainBench / BrainGPT Nature 2024</u>](https://www.nature.com/articles/s41562-024-02046-9),[ <u>Question Generation 2601.22444</u>](https://arxiv.org/abs/2601.22444),[ <u>AIA Forecaster 2511.07678</u>](https://arxiv.org/abs/2511.07678) (no-finetuning baseline). Blog:[ <u>Training LLMs to Predict World Events (Mantic / Thinking Machines)</u>](https://thinkingmachines.ai/news/training-llms-to-predict-world-events/).

* **Consensus: outcome RL on small open-weights models reaches prior-generation closed-weights parity, not current frontier.** Turtel et al.'s[ <u>DPO + LoRA on Phi-4 14B and DeepSeek-R1 14B</u>](https://arxiv.org/abs/2502.05253) (LoRA rank 16, alpha 32, dropout 0.05) trained on 9,800 Polymarket questions and 18,854 self-play traces tied GPT-4o (Brier 0.197 vs 0.196, adjusted p > 0.7) on 2,300 questions resolving Dec 25 2024 to Jan 23 2025, but o1 and Claude 3.5 Sonnet were already ahead by Feb 2025. Lightning Rod's[ <u>Outcome-based RL 14B</u>](https://arxiv.org/abs/2505.17989) extends to a ReMax loop and ties o1. Their[ <u>Foresight-32B via Future-as-Label</u>](https://arxiv.org/abs/2601.06336) beats Qwen3-235B prompting (Metaculus Brier 0.179 vs 0.211), where Qwen3-235B was never a top-2 closed model. Max Planck's[ <u>OpenForecaster 8B</u>](https://www.alphaxiv.org/abs/2512.25070) trained on 52,000 CCNews-derived pastcast questions competes with prior-generation 100B+ closed peers on FutureX, below current leaders. The closed baseline in every comparison is a strong-but-not-current model.
* **Consensus: self-play with resolved outcomes is the working training signal.**[ <u>Halawi et al.'s SSFT pipeline</u>](https://arxiv.org/abs/2402.18563) (system Brier 0.179 vs crowd 0.149; system+crowd 0.146) is the foundational version. Turtel's DPO ranking by proximity to outcome ([<u>2502.05253</u>](https://arxiv.org/abs/2502.05253)), Lightning Rod's ReMax-based outcome RL ([<u>2505.17989</u>](https://arxiv.org/abs/2505.17989)), Future-as-Label's RLVR on delayed real-world outcomes ([<u>2601.06336</u>](https://arxiv.org/abs/2601.06336)), and Max Planck's GRPO with combined Brier + accuracy reward ([<u>2512.25070</u>](https://www.alphaxiv.org/abs/2512.25070)) are all fully autonomous training loops that only need a stream of resolved questions. Outcome supervision beats human-written rationale chains in every captured comparison.
* **Consensus: auto-generated questions remove the human-curation bottleneck.** Bosse et al.'s[ <u>Question Generation paper</u>](https://arxiv.org/abs/2601.22444) reports 96% validity and 95% resolution accuracy on automatically created questions. OpenForecaster's 52,000-question OpenForesight set (built from CCNews) is the largest captured demonstration that the loop runs at scale without human curation. The 3 production data sources for finetuning are Polymarket past questions, Metaculus past questions, and auto-generated CCNews questions; once the question pipeline works, a team of 1 to 2 can run training without external annotation.
* **Disagreement: open-weights finetune vs current frontier single-model quality is split.** Mantic's RL finetune of gpt-oss-120b via[ <u>Thinking Machines' Tinker API</u>](https://thinkingmachines.ai/news/training-llms-to-predict-world-events/) crossed past Gemini 3 Pro on Mantic's internal Metaculus baseline-points scale (38.6 to 45.8), suggesting a 100B-class open-weights finetune can match a current closed leader on at least one test.[ <u>AIA Forecaster</u>](https://arxiv.org/abs/2511.07678) reaches superforecaster parity (Brier 0.108 vs 0.111 across 2 ForecastBench rounds) with no finetuning at all on a retrospective / backtested evaluation, suggesting scaffolding plus calibration on a frontier API can get to the same place provided the backtest hygiene holds up. Mantic's production 5-model ensemble keeps Gemini 3 Pro, GPT-5, and Grok 4 alongside the fine-tune, so even where the finetune crosses the frontier on an internal test the deployed configuration uses the finetune for error decorrelation, not replacement.
* **Operating consensus: finetune for cost and decorrelation, scaffold for current-frontier quality.** Open-weights bot at 7B to 32B with 1 to 4 weeks of effort: finetune via DPO or outcome RL on a self-play + resolved-outcome stream ([<u>Turtel 2502.05253</u>](https://arxiv.org/abs/2502.05253),[ <u>Foresight Learning 2601.06336</u>](https://arxiv.org/abs/2601.06336),[ <u>OpenForecaster 2512.25070</u>](https://www.alphaxiv.org/abs/2512.25070)); unit economics beat per-token frontier API calls especially across MiniBench rounds and seasonal tournaments. Frontier API customer: skip finetuning, put the same budget into agentic search, ensembling 3 to 7 runs, Platt scaling, and news-source breadth ([<u>AIA Forecaster</u>](https://arxiv.org/abs/2511.07678)). Mixed strategy (Mantic's deployed configuration): use a frontier closed model and a finetuned open-weights model as the most-decorrelated ensemble member.

## **Gaps in this review's scope**

Things the input set does not cover well:

* Financial-forecasting-specific bots and their benchmarks are only touched via the trading arenas. There is a much larger industry literature around time-series foundation models (Chronos, Moirai, TimeGPT) that is out of scope here and worth a separate pass.
* Non-English forecasting. Every captured source is English-language. There are active forecasting communities on Chinese and Russian-language platforms that are not represented.
* Prediction-market mechanism design (AMMs, LMSR, batch auctions).[ <u>Tell Me Why</u>](https://arxiv.org/abs/2502.13410) is the only captured paper that touches aggregation-mechanism design.
* Multimodal forecasting beyond the 2 Nature papers. Video, image, and audio signals are mostly absent from the captured literature.

None of these review-scope gaps are reasons to doubt the substantive findings above, but they do bound the scope of the claims this review can make.