# Main Takeaways
**Top Finding:**
* **The number of research sources is the strongest predictor of forecasting accuracy**: In the Fall 2025 bot-maker survey, the number of distinct research sources used correlated with tournament score at r \= 0.42 (p \= 0.006). Winners used an average of 1.72 research sources compared to 1.00 for non-winners. Common winning stacks included AskNews + Perplexity + a custom scraper, or AskNews + Exa + OpenAI web search — the pattern is diversity, not brand loyalty. Earlier seasons did not measure source count directly, but their consistent finding that no individual provider stood out is compatible with this interpretation.
**Other Findings:**
* **We have not found evidence that any single search provider outperforms the others**: Across three seasons of Metaculus's AI benchmarking series (Q1 2025, Q2 2025, and Fall 2025), no provider has shown a statistically significant advantage. AskNews, Perplexity, and Exa have each performed competitively in different contexts, but confidence intervals overlap heavily. However, these comparisons could benefit from a larger sample size, and we cannot rule out that meaningful differences exist.
* **AskNews was widely used but was not associated with higher scores**: In the Fall 2025 survey, AskNews usage rates were identical between winners (38%) and non-winners (40%). In Q2, AskNews users actually scored slightly lower on average, though the difference was not statistically significant given the small sample.
* **Now that most competitors use frontier models, research and scaffolding are the differentiators**: By Fall 2025, 34 of 39 survey respondents used a frontier model.¹ The gap between the best and worst scaffolded bots using the *same* foundation model was roughly 9,500 score points² — comparable to the gap between frontier and non-frontier models.
# Introduction
Over the past year, Metaculus has run a series of AI benchmarking tournaments, pitting AI forecasting bots against each other and against human Pro Forecasters on hundreds of real-world prediction questions. Starting in Q1 2025, Metaculus ran dozens of in-house "Metac Bots" — bots that use the same prompt but vary by LLM model and search provider — to serve as controlled baselines.
One recurring question from bot-makers and researchers alike has been: **does the choice of search provider matter for forecasting accuracy?** Bots need current information to forecast well, and the way they retrieve that information — whether through [AskNews](https://asknews.app/), [Perplexity](https://www.perplexity.ai/), [Exa](https://exa.ai/), web scraping, or other tools — could plausibly make a significant difference.
This post synthesizes what we have learned across three seasons of tournament data (Q1 2025, Q2 2025, and Fall 2025), over 60 survey responses from bot-makers, and controlled comparisons between Metac Bots using different search providers.
**Scoring:** All scores in this post are [Spot Peer Scores](https://www.metaculus.com/help/scores-faq/#peer-score). The Peer Score compares a forecaster's accuracy to all other participants on each question — the sum of all participants' Peer scores is always 0, and if all participants score a 0, they are all equally skilled. A "spot" score evaluates only the prediction standing when the question closes, rather than averaging over the question's lifetime. Tournament rankings are determined by the sum of spot peer scores across all questions. Importantly, the Peer Score is a [proper scoring rule](https://www.metaculus.com/help/scores-faq/#proper-scoring): forecasters maximize their score by predicting their true beliefs.
**A note on statistical power:** The survey samples in this analysis range from 11 (Q1) to 39 (Fall 2025), with non-winner comparison groups as small as 9. With samples this small, many real effects will fail to reach statistical significance (conventionally p \< 0.05). Throughout this post, a non-significant result should not be read as "this doesn't matter" — it often means "we don't have enough data to tell." We flag significance where we can compute it, but encourage readers to weigh effect sizes and consistency across seasons alongside p-values.
# Q1 2025: First Controlled Comparison
[Tournament](https://www.metaculus.com/tournament/aibq1/) | [Analysis](https://www.metaculus.com/notebooks/38673/q1-ai-benchmarking-results/)
In Q1, Metaculus ran 11 in-house Metac Bots, all using the same prompt. Three of these used OpenAI's o1 as the forecasting model but varied only in their search provider:
| Search provider | Rank | Avg Score | Sum of Score |
| :-------------------------------------- | ---: | --------: | -----------: |
| AskNews (metac-o1) | 1st | 7.99 | 3,242 |
| Perplexity Sonar-Pro (metac-perplexity) | 4th | 2.08 | 793 |
| Exa custom wrapper (metac-exa) | 6th | -0.30 | -119 |
From this, AskNews appeared to be a better news source for bot forecasting. However, the 95% confidence intervals overlapped considerably, meaning we could not rule out that the differences were due to chance.
In the Q1 bot-maker survey (11 respondents), Perplexity was the most popular research tool (72.7% of respondents), followed by static web scraping (45.5%) and AskNews (36.4%). The survey did not collect enough data to draw conclusions about which provider helped most.
# Q2 2025: Expanded Comparison
[Tournament](https://www.metaculus.com/tournament/aibq2/) | [Analysis](https://www.metaculus.com/notebooks/40456/q2-ai-benchmark-results/)
In Q2, Metaculus scaled up the experiment significantly, running 42 in-house Metac Bots across many model-and-search combinations. This allowed more granular comparisons.
**DeepSeek-R1 with different search providers:**
Multiple Metac Bots paired DeepSeek-R1 with different search providers:
| Search provider | Rank | Avg Score | Sum of Score |
| :----------------------------- | ---: | --------: | -----------: |
| Perplexity Sonar-Reasoning-Pro | 13th | 1.83 | 610 |
| Perplexity Sonar | 15th | 1.50 | 501 |
| GPT-4o-search-preview | 16th | 1.35 | 453 |
| AskNews | 17th | 1.15 | 390 |
| Perplexity Sonar-Pro | 18th | 1.04 | 347 |
| Exa Smart Searcher | 28th | -1.49 | -466 |
**Gemini 2.5 Pro with different search providers:**
Multiple Metac Bots paired Gemini 2.5 Pro with different search providers:
| Search provider | Rank | Avg Score | Sum of Score |
| :----------------------------- | ---: | --------: | -----------: |
| AskNews | 7th | 5.29 | 1,768 |
| Perplexity Sonar-Reasoning-Pro | 12th | 2.27 | 741 |
| Exa Pro | 19th | 0.89 | 281 |
For Gemini 2.5 Pro, AskNews performed best; for DeepSeek-R1, Perplexity's Sonar-Reasoning-Pro came out ahead with AskNews placing fourth. In both cases, confidence intervals overlapped substantially. The Q2 analysis concluded: **"Best search is uncertain."**
In the Q2 bot-maker survey (19 respondents), AskNews was the most commonly used research tool (11 of 19, or 58%). The analysis compared coverage-adjusted scores of AskNews users vs. non-users and found a difference of -432 points (95% CI: -1,682 to +818) — not statistically significant, with the wide confidence interval reflecting the small sample. The point estimate was actually slightly in favor of non-AskNews users.
# Fall 2025: Breadth Over Brand
[Tournament](https://www.metaculus.com/tournament/fall-aib-2025/)
The Fall 2025 tournament was the largest to date, with 123 bots competing over four months. Metaculus ran an expanded set of Metac Bots and conducted a more comprehensive post-tournament survey (39 respondents: 29 winners, 10 non-winners).
## Metac Bot Comparisons
The Fall leaderboard includes several DeepSeek-R1 bots paired with different search providers:
| Search provider | Rank | Avg Score³ | Sum of Score |
| :----------------------------- | -----: | ---------: | -----------: |
| GPT-4o-search-preview | 26th | 3.85 | 1,283 |
| XAI livesearch | 31st | 2.06 | 680 |
| Perplexity Sonar-Reasoning-Pro | 33rd | 1.46 | 485 |
| Perplexity Sonar-Pro | 34th | 0.57 | 189 |
| Perplexity Sonar | 54th | -2.75 | -911 |
| Perplexity Sonar-Reasoning | 57th | -3.30 | -1,092 |
| AskNews | 58th | -3.38 | -1,110 |
| *No research* | *65th* | *-7.07* | -2,345 |
| Exa Online | 66th | -7.79 | -2,542 |
GPT-4o-search-preview led by a wide margin, while AskNews placed near the bottom — a reversal from Q2 where AskNews ranked mid-pack with DeepSeek-R1. Also notable: the no-research bot (which did no search at all) outscored some search-equipped variants, suggesting that low-quality research can actively hurt performance.
A notable new entrant was **metac-asknews-deepnews\[research-only],** which scored 3,491 — the highest-scoring Metac Bot on the Fall leaderboard. This bot used AskNews's newer DeepNews research product rather than the standard AskNews API. For reference, the top AskNews-based model bot was metac-gpt-5+asknews at 3,252. While comparisons between bots using different models are harder to interpret, the strong showing of this bot suggests that deeper, more comprehensive research tools may provide an edge.
Other search-relevant bots on the Fall leaderboard include metac-grok-4-live-search\[research-only-bot] (score 2,336) and metac-gemini-2-5-pro-grounding\[research-only] (score -1,037).
These comparisons are suggestive but we lack per-comparison error bars.
## The Survey Finding: Number of Sources Is What Matters
The strongest finding from the Fall 2025 survey is about research **breadth**, not provider choice. The number of distinct research sources used by a bot correlated with tournament score at r \= 0.42 (p \= 0.006).
Winners pulled from an average of 1.72 distinct research sources; non-winners from 1.00.⁴
Meanwhile, usage of any *individual* provider showed no statistically significant association with winning, though some differences in adoption rates are suggestive:
| Provider | Winner use % | Non-winner use % | p |
| :------------------ | -----------: | ---------------: | ----: |
| OpenAI web search | 41% | 20% | 0.279 |
| Perplexity | 38% | 10% | 0.131 |
| AskNews | 38% | 40% | 1.000 |
| Static web scraping | 21% | 0% | N/A |
| Exa | 17% | 10% | 1.000 |
Put simply: the takeaway isn't which source to pick — it's that one source is usually not enough.
Common research stacks among top-performing bots included combinations like AskNews + Perplexity + a custom scraper, or AskNews + Exa + OpenAI web search.
## Other Research-Related Findings from the Fall Survey
The Fall survey also revealed several findings about research *strategy* that go beyond provider choice but are relevant context for bot-makers thinking about their research pipeline.
**Agentic search over one-shot retrieval:** Multiple top bot-makers described moving from single searches to multi-step pipelines. As one winner put it:
> I'm not convinced that one-shot research steps can be competitive with the more agentic/adaptive search based approaches. Humans look things up, find more questions to ask, and then look more things up. Bots should behave the same way.
Another described their loop:
> A form of dynamic search (like agentic search, or some form of being able to investigate further) is important. My loop involved generating queries, searching, checking results, and generating new queries to address remaining information gaps.
However, not all bot-makers found agentic architectures helpful. One cautioned:
> Agents are overrated. I tried them, and ended up going back to traditional pipelines. Using LLMs to combine distributions was silly and kept driving me insane and I should've moved away from that earlier.
This last complaint is specifically about using LLMs for agentic *reasoning* (combining probability distributions), not agentic *search*. The distinction matters: iterative search to fill information gaps appears in several winning bots, but wrapping every step in an agent framework is not necessarily better than a well-structured traditional pipeline.
**More LLM calls, split across pipeline stages:** Winners averaged \~28 LLM calls per question vs. 7 for non-winners (p \= 0.022), typically splitting their pipeline into research, forecast, critique, and aggregation steps rather than a single prompt.
**Historical question lookup:** 34% of winners had their bot look up similar previously resolved questions on Metaculus or prediction markets to use as a reference point when forecasting. Zero non-winners did this (p \= 0.04). Using historical forecasting data — not just current news — as an input appears to be a distinguishing strategy among top bots.
**Research vs. reasoning balance doesn't matter:** When asked whether they optimized more for research (external information retrieval) or reasoning (processing information given to the LLM), winners and non-winners reported nearly identical distributions across the spectrum. The correlation with score was weak and non-significant. Winners appear to do both competently rather than picking one at the expense of the other.
# Summary Across Seasons
| Season | Key search finding | Evidence type |
| :-------- | :--------------------------------------------------------------------------------------------------- | :----------------------------------------- |
| Q1 2025 | AskNews > Perplexity > Exa with o1, but not significant | 3 controlled Metac Bot comparisons |
| Q2 2025 | "Best search is uncertain"; AskNews performs well in some pairings but no significant overall effect | 42 Metac Bots + 19-person survey |
| Fall 2025 | Number of sources (r\=0.42, p\=0.006) is the strongest predictor; no individual provider significant | 39-person survey + leaderboard comparisons |
The pattern across seasons is suggestive: no individual provider has emerged as a clear winner, and the Fall 2025 data — the largest and most rigorous survey to date — points toward **research breadth and strategy** rather than provider choice as the key variable. That said, this breadth finding has only been directly measured once, so it will be important to see whether it replicates in future seasons.
# Practical Recommendations
We group these by strength of evidence.
**Strongest evidence (p \< 0.01):**
1. **Use multiple research sources.** The number of distinct sources used correlated with tournament score at r \= 0.42 (p \= 0.006) in the Fall 2025 survey. Combining two or three providers — such as AskNews + Perplexity + a web scraper — is a common pattern among winners.
**Anecdotal (based on individual examples or quotes, not statistical tests):**
1. **Consider agentic search.** Several top bot-makers describe multi-step search pipelines that generate queries, evaluate results, and search again. However, not all found this helpful — traditional pipelines can also work well.
2. **Be cautious with low-quality research.** On the Fall leaderboard, DeepSeek-R1 with no search outscored some search-equipped variants, suggesting that poor-quality research can actively hurt.
3. **Provider performance may vary by model.** AskNews ranked well with Gemini 2.5 Pro in Q2 but poorly with DeepSeek-R1 in Fall. If you switch models, re-evaluate your search setup.
# Caveats
Several important caveats apply to these findings:
* **Small sample sizes.** Survey response counts ranged from 11 (Q1) to 39 (Fall). The non-winner comparison group in Fall was only 9-10 respondents (see footnote 3). With samples this small, we cannot compute reliable error bars for many comparisons, and individual p-values should be treated as suggestive rather than definitive.
* **Multiple comparisons.** Across the surveys, we tested many features against score. With \~15 binary features and 6 continuous features tested in the Fall survey, some "significant" results are expected by chance. A Bonferroni-corrected significance threshold would be \~0.002. No individual finding in the Fall survey survives this correction, including the research-breadth result (p \= 0.006).
* **Correlation, not causation.** Bot-makers who use multiple research sources may also invest more time, effort, and thought into other aspects of their bot. The additional sources may not be the causal mechanism — they may be a proxy for overall engineering effort.
* **Selection bias.** Survey respondents are self-selected. Winners were required to respond to receive prizes, but non-winners who responded may skew toward the more engaged end of the spectrum, potentially understating the true gap.
* **Self-report.** What bots actually did is inferred from survey responses, not cross-verified through code inspection (except for a small number of bots that submitted code for review).
* **Research formatting is not controlled for.** Different providers deliver research in different formats: AskNews returns a list of article summaries with URLs, Perplexity returns an LLM-styled narrative response, and some providers do hidden processing in the background. These formatting differences may interact with the forecasting model in ways that affect score independently of information quality.
* **Provider capabilities change over time.** Search providers update their products between seasons. A provider that performed well in Q1 may have different capabilities by Fall 2025. Our findings reflect the state of these tools during the relevant tournament periods.
***
# Footnotes
¹This count depends on how "frontier" is defined. Using a narrower definition (GPT-5, Claude 4.x, Gemini 2.5 Pro, Grok 4, o3), 34 of 39 respondents qualify. A broader "strong" tier that also includes DeepSeek R1 and the full o3 family yields 36-37 of 39. Either way, model choice has largely converged among competitors.
²This spread (approximately 9,488 points) is measured between the highest- and lowest-scoring bots listing GPT-5 as a primary model. Several of these bots used additional models alongside GPT-5, so the spread reflects scaffolding differences broadly, not a pure single-model comparison.
³Fall 2025 average scores are estimated by dividing the sum of scores by the number of questions derived from the leaderboard coverage percentage (coverage × 377). All DeepSeek-R1 bots answered approximately 326-333 questions.
⁴The Fall 2025 survey collected 39 total responses (29 winners, 10 non-winners). For the winner vs. non-winner comparison, minibench-only winners were reclassified as non-winners, yielding groups of n\=28 and n\=9. The Pearson correlation (r\=0.42, p\=0.006) was computed across all 39 respondents.