*By Ben Wilson and John Bash* # Main Takeaways **Top Findings:** * **Pro forecasters significantly outperform bots**: Our team of 10 [Metaculus Pro Forecasters](https://www.metaculus.com/pro-forecasters/) demonstrated superior performance compared to the top-10 bot team, with strong statistical significance (p = 0.00001) based on a one-sided t-test on Peer scores. * **The bot team did not improve significantly in Q2 relative to the human Pro team**: The bot team’s [head-to-head score](https://www.metaculus.com/notebooks/28552/comparing-forecasting-track-records-for-ai-benchmarking-and-beyond/#:~:text=Our%20approach%20is%20to%20calculate%20Peer%20scores%20as%20if%20there%20were%20a%20tournament%20with%20only%20two%20participants%3A%20an%20individual%20bot%20and%20the%20human%20aggregate.%20We%20refer%20to%20this%20as%20the%20Head%2Dto%2Dhead%20score.) against Pros was -11.3 in [Q3 2024](https://www.metaculus.com/notebooks/28784/aibq3results/) (95% CI: \[-21.8, -0.7]), then -8.9 in [Q4 2024](https://www.metaculus.com/notebooks/35291/q4-ai-benchmarking-results/) (95% CI: \[-18.8, 1]), then -17.7 in [Q1 2025](https://www.metaculus.com/notebooks/38673/q1-ai-benchmarking-results/#bot-team-vs-pro-team) (95% CI: \[-28.3, -7.0]), and now -20.03 \[-28.63, -11.41] with no clear trend emerging. (Reminder: a lower head-to-head score indicates worse relative accuracy. A score of 0 corresponds to equal accuracy.) **Other Takeaways:** * **This quarter’s winning bot is open-source**: Q2 Winner [Panshul](https://www.linkedin.com/in/panshul-sharma/) has very generously made his bot [open-source](https://github.com/Panshul42/Forecasting_Bot_Q2). The bot writes separate [“outside view” and “inside view”](https://www.lesswrong.com/w/inside-outside-view) reports, using a 6-7 step agentic approach, with Serper to run Searches and BrightData.com to scrape HTML. It then runs five final predictions using sonnet 3.7 twice (later sonnet 4), o4-mini twice, and o3 once. * **The best model was OpenAI’s o3**: Metaculus ran many bots that used the same prompt but different LLMs. The one using OpenAI’s o3 finished first among these, and 2nd overall in the tournament. * **Models matter more than prompts**: Metaculus’s simple in-house Metac Bots have continued to perform well. The Metac Bots that use the most up-to-date models have continually ranked in the top 6 in the last 4 quarters, and the best one placed 2nd this quarter. This implies that the most important factor for AI forecasting is the underlying model, more than scaffolding. These bots do a single internet search, run a simple prompt, forecast 5 times with that prompt, and submit an aggregation of these forecasts. * **Students and hobbyists performed well**: Excluding Metaculus’s in-house Metac Bots, the top 3 bot-makers were hobbyists or students. Notably, they came just ahead of the startups and other commercial entities that competed. * **Aggregation and iteration are key**: Based on the results of our bot maker survey, best practices include aggregating multiple forecasts and manually reviewing the bot’s reasoning on past questions to improve prompts and scaffolding for the next questions. # Introduction In the second quarter of 2025, we ran the fourth tournament in our AI Benchmarking Series, which aims to assess how the best AI forecasting bots compare to the best humans on real-world forecasting questions, like those found on Metaculus. Over the quarter, 54 bot-makers competed for \$30,000 by forecasting 348 questions. Additionally, a team of ten Pro Forecasters forecasted 96 of those 348 questions to establish a top human benchmark. Questions spanned many topics, including technology, politics, economics, the environment, and society. They covered different types of outcomes: Binary (“Yes”/”No”), Numeric (e.g., “\$4.6M”, “200 measles cases”, etc.), and Multiple Choice (e.g., “Max Verstappen”, “AAPL”, etc.). See a full overview of Metaculus’s AI forecasting benchmark at the tournament [home page](https://metaculus.com/aib) and [resource page](https://www.metaculus.com/notebooks/38928/ai-benchmark-resources/). # Methodology In order to test how well bots are doing relative to each other and relative to humans, we set up a forecasting tournament for bots and one for Pros. The Pro forecasts were hidden from bots to prevent them from being copied by bots. Each tournament launched a series of questions that were resolved sometime during the quarter, and asked participants to assign probabilities to outcomes. We then used our established [scoring rule](https://www.metaculus.com/help/scores-faq/#peer-score) to evaluate participants. During the analysis, we aggregate the predictions of groups of forecasters to create ‘teams’. **Data Collection**: * **Bot Tournament**: 54 bot makers and 42 in-house Metac Bots (running different LLM models) participated in a tournament of 348 questions that were not [annulled](https://www.metaculus.com/faq/#ambiguous-annulled). Each week, we launched a set of \~50 questions, one at a time, every 2 hours. These questions stayed open for 2 hours, then closed before the next one launched. This ensured forecasting occurred at all times during the day (including night hours). Bots submitted their forecasts using our [API](https://www.metaculus.com/api/). The tournament consisted of [binary](https://www.metaculus.com/faq/#question-types) (58%), [numeric](https://www.metaculus.com/faq/#question-types) (21%), and [multiple-choice](https://www.metaculus.com/faq/#question-types) (21%) questions. Questions spanned an extensive range of topics, including geopolitics, economics, elections, sports, tech, etc. Botmakers were allowed to update their bots over time (e.g., incorporate a newer model, or iterate on prompts), but metac-bots stayed the same over the full quarter. More binary questions were resolved “No” (80%) than “Yes” (20%). This is common in the forecasting field; for example, 70% of Metaculus’s main feed questions resolve to “No.” * **Pro Tournament**: 10 Pro forecasters predicted 96 (plus 4 that were [annulled](https://www.metaculus.com/faq/#ambiguous-annulled)). Three of those had match-over problems, leaving 93 scorable questions with direct comparisons between bots and Pros. [Metaculus selects Pro Forecasters](https://www.metaculus.com/pro-forecasters/) for their excellent forecasting track record and their history of clearly describing their rationales. Pro questions were randomly sampled from each week’s bot questions (with some constraints to space out schedules). Pros had 2 days to forecast each question before it closed. The Pro tournament was private and hidden throughout the tournament, which prevented bots from copying the Pros’ forecasts. Pros, however, could see each other’s forecasts and could communicate in the comment section of each question. * **Metac Bots**: Metaculus ran 42 in-house Metac Bots. Each used the same prompt but used a different LLM to forecast, and occasionally a different method for doing research. In general, these bots do an internet search, run a \~30-line prompt, forecast 5 times with that prompt, and submit an aggregation of these forecasts. Each bot’s name indicates which model and research provider it used. For instance, “metac-gpt-4o+asknews” uses GPT-4o as a model and AskNews as a research source. **Scoring**: For a deeper understanding of scoring, we recommend reading our [scoring FAQ](https://www.metaculus.com/help/scores-faq/). We primarily use spot peer scores and head-to-head scores in this analysis. * **Peer Score**: The Peer score compares the relative performance of a forecaster (or team) vs. all others on a question, over the whole question lifetime. Unlike Baseline or Brier scores, it adjusts for the difficulty of a question based on the performance of others. The score is zero-sum (i.e., the sum of all participants’ peer scores is 0). A higher score indicates greater relative accuracy. A score of 0 corresponds to average accuracy. For reference, the average Peer score for the Metaculus Community Prediction (a recency-weighted median of all forecasters) is 12.9. The community prediction has ranked in the top 10 on the global site leaderboard over every 2-year period since 2016. So, a score of 12.9 is a very good peer score when competing against Metaculus users. This is a decent comparison point for the AI tournament. However, it's worth noting that the bots are not “the average Metaculus user” and a good score in the AI tournament is different than on the actual site due to the skill level of competitors. * **Spot Peer Score**: Spot Peer scores are the same as peer scores, except evaluated only at a single point in time, using the forecasters’ standing predictions at that time (rather than averaging scores over the question lifetime). On the main Metaculus site, spot peer scores are often evaluated when the community prediction is revealed to participants. For the bot and Pro tournaments, Spot Peer scores were evaluated when questions closed. * **Head-to-Head Score**: Head-to-head scores are the same as spot peer scores, except they compare only two participants. For example, if comparing the bot team and Pro team, the head-to-head score is $100 \times \ln(\text{bot team prediction} / \text{pro team prediction})$ (for a question that resolves Yes). * **Weights**: Some bot questions were given weights in order to counteract possible correlated resolutions between questions. This doesn’t apply to pro/quarterly cup questions, which all have a weight of 1. If you see below a score for a bot tournament question, then that score has been multiplied by a weight between 0 and 1 inclusive. **Team selection and aggregation**: In the analyses below, we sometimes aggregate a group of forecasters to create a “team”. To aggregate predictions, we take the median for binary predictions, the normalized median for multiple choice predictions, and an average (aka mixture of distributions) for numeric predictions. This is the same as the default aggregation method for the Community Prediction shown on Metaculus, except the Community Prediction also weighs predictions by recency. Note that occasionally a member of a team misses a question, and thus is not included in the team aggregate prediction for that question. Below is how the compositions of the teams were decided: * **Bot Team**: We took the 348 tournament questions, then removed the 93 questions that were also asked to the pros, to get 255 bot-only questions. We compute the bots’ Spot Peer Scores on these questions, and rank the bots in a leaderboard by the sum of their scores. The top 10 bots from this leaderboard are our bot team. We exclude the Pro questions to avoid look-ahead bias in the Pro-bot comparison. By using the bot-only questions to determine the team, we effectively run a “qualification tournament” to choose bot team members (similar to how the pros were chosen based on their performance on past tournaments). * **Pro Team**: The Pro team includes all 10 pros who participated in the Pro tournament. **Miscellaneous notes on Data Collection**: * **Very bad numeric forecasts can be rejected**: Numeric questions have an upper and lower bound that forecasters on the site are generally expected to forecast within. To ensure numerical stability (and non-astronomically-negative scores), our API rejects predictions that put no or very little probability mass in bounds. This quarter, the prompt used for Metaculus’s in-house Metac Bots did not include a mention of what the upper and lower bounds are. Occasionally, some “Metac Bots” would forecast out of bounds in this way (especially those running lower-quality models), and it is expected that some participants encountered this error as well. Our forecasting user interface does not let humans make forecasts that bad either, so in this sense, they’re on equal footing, but it does mean that bots are not punished for their very worst numeric forecasts (we’ll see that they still largely underperform humans on numeric questions anyway). * **o3-mini and o3-mini-high ran on the same account**: By mistake, the in-house Metac Bots running o3-mini and o3-mini-high were both using the same account on the site (i.e., o3-mini-high’s account). We suspect that since o3-mini-high takes longer to forecast, o3-mini-high made the last forecast on each question. Thus, this account most likely represents o3-mini-high’s forecasts generally. However, we aren’t able to confirm this easily. * **Other small data anomalies**: 1) Each metac bot missed some questions on the site due to various factors like provider outages, poorly formatted responses, etc. 2) There was a bug in the multiple choice parsing code that a large percentage of competitors and metac-bots were using. We relaunched the 8 affected questions. 3) For Metac Bots that did not use AskNews and instead used a research-enabled LLM to get their research report (e.g., Perplexity), the full question JSON was passed to the research LLM’s prompt. In previous quarters, only the question text was passed to the prompt. This analysis generally follows the [methodology](https://www.metaculus.com/notebooks/28552/comparing-forecasting-track-records-for-ai-benchmarking-and-beyond/) we laid out at the beginning of the tournament series, but now uses a simplified bot team selection algorithm we started using in Q1 (as described above). # How do LLMs Compare? To compare the capabilities of different LLM models, Metaculus ran 42 bots, all with the same prompt. This prompt has stayed generally the same over all four quarterly benchmarks. There were some minor updates to the prompt in Q1 to support numeric and multiple-choice questions. See the section on code/data for links to the template bot code and prompts. Using the results from the benchmark, we simulated a tournament where these bots competed only against each other (no other participants). This tournament took all the questions from the bot tournament and removed the forecasts of non-metac-bot participants. The bots’ spot peer scores and 95% CI against each other are shown below. The first graph shows the top 12 bots, while the second is zoomed out to show all participants.

Each bot’s name indicates which model and research provider were used. For instance, metac-gpt-4o+asknews uses GPT-4o as a model and AskNews as a research source. You can see code for each model in the “Code and Data” section of this piece. * **OpenAI’s o3 is the best model**: The bot powered by o3 significantly outperformed even the next best non-o3 model. It’s possible that the prompt worked better for o3 than other models, but some of our [other research](https://www.metaculus.com/notebooks/38421/automated-prompt-engineering-for-forecasting/) shows that prompt engineering has less of an impact with stronger models. * **Cost-effectiveness goes to o4-mini and DeepSeek-R1**: The most cost-effective models would be o4-mini-high and DeepSeek-R1. * **Best search is uncertain**: We benchmarked different search providers with both Gemini-2.5-Pro and DeepSeek-R1. Only a few variations were tested with Gemini-2.5-Pro, and more were tested with DeepSeek-R1. The best search model for Gemini-2.5-Pro was AskNews, with Perplexity’s Sonar-Reasoning-Pro in second. The best search model for DeepSeekR1 was Perplexity’s Sonar-Reasoning-Pro with Perplexity’s Sonar in second and AskNews in third. Most of these bots have score error bars that overlap a lot, so we can’t say much about which model is best. * **Thinking is a mixed bag**: It seems that adding more reasoning tokens sometimes helps forecasting and sometimes does not. In favor of thinking, o1 with reasoning set to “high” did better than regular o1. Sonnet 3.7 also did better with thinking than without. However, OpenAI’s o3 with reasoning set to “high” did worse than regular o3. # Which Bot Strategy is Best? Let's take a look at the bot-only tournament. This tournament pits all the bots (both bot-makers and in-house Metac Bots) against each other and does not include any humans. You can see the full leaderboard [here](https://www.metaculus.com/tournament/aibq2/). Since there are 96 bots, let's first focus on the bots that topped the leaderboard. Below are the average scores of the top 10 bots when ranked by the sum of their spot peer scores. We rank by sum of spot peer scores to reduce the noise caused by bots that got high averages by getting lucky on only a few questions.

The best bot was made by bot maker Panshul42. **Panshul has very generously made his bot open-source. You can review his code and prompts in full** [**here**](https://github.com/Panshul42/Forecasting_Bot_Q2)**.** We interviewed [Panshul](https://www.linkedin.com/in/panshul-sharma/) about his bot and asked for his description of how it forecasts questions. The bot starts by running multiple research reports. One report specifically looks for the outside view (historical rates), while the other focuses on the inside view (situation-specific factors). Originally, he ran these reports using Perplexity and deepnews, but switched to using a 6-7 step agentic approach that included using Serper to run Google Searches and [BrightData.com](http://brightdata.com) to scrape HTML. He then runs 5 final predictions using sonnet 3.7 twice (later sonnet 4), o4-mini twice, and o3 once. In development, Panshul mostly manually reviewed his bot’s outputs and used his judgment to improve things. However, he also benchmarked his bot’s predictions against the community prediction (e.g., see [here](https://github.com/Metaculus/metac-bot-template/tree/main?tab=readme-ov-file#early-benchmarking) and [here](https://github.com/Panshul42/Forecasting_Bot_Q2/blob/318499e284fba07396550677b152c35f7e47f255/Bot/benchmark.py)) on Metaculus using \~35 questions each time. The second-best bot was one of Metaculus’s template bots, metac-o3. metac-o3 used AskNews to gather news and OpenAI’s o3 model to forecast. Links to its code are included in the “links and code” section. Here is a [link](https://www.metaculus.com/questions/37627/will-donald-trump-and-volodymyr-zelenskyy-speak-on-the-phone-with-each-other-before-june-15-2025/#comment-328273) to an example of its research and reasoning. For reference, here is the prompt that metac-o3 (and the other metac bots) used for binary questions: """ You are a professional forecaster interviewing for a job. Your interview question is: \{question\_title} Question background: \{paragraph\_of\_background\_info\_about\_question} This question's outcome will be determined by the specific criteria below. These criteria have not yet been satisfied: \{paragraph\_defining\_resolution\_criteria} \{paragraph\_defining\_specifics\_and\_fine\_print} Your research assistant says: \{many\_paragraphs\_of\_news\_summary} Today is \{today}. Before answering you write: (a) The time left until the outcome to the question is known. (b) The status quo outcome if nothing changed. (c) A brief description of a scenario that results in a No outcome. (d) A brief description of a scenario that results in a Yes outcome. You write your rationale remembering that good forecasters put extra weight on the status quo outcome since the world changes slowly most of the time. The last thing you write is your final answer as: "Probability: ZZ%", 0-100 """ Our template bots have reached the top 10 in all four tournaments over the last year. metac-gpt-4o (fka mf-bot-1) placed 4th in Q3 2024, metac-o1-preview (fka mf-bot-4) placed 6th in Q4 2024, and metac-o1 placed 1st in Q1 2025. metac-gpt-4o’s ranking has been sliding over time, as better models enter the competition, getting 17th of 44 in Q4 2024, 44th out of 45 in Q1 2025, and 82nd out of 96 in Q2 2025. Most participants did not use o3 in their forecasting during Q2 due to its cost (or at least when we offered LLM credits, few requested it). **From the above, we conclude that the most important factor for accurate forecasting is the base model used, with additional prompting and infrastructure providing marginal gains.** Here is the zoomed-out full leaderboard. Like the chart above, this graph is ordered by the sum of spot peer scores, which is used to determine rankings on the site (i.e., so the 1st bar shown took 1st place), and the y-axis shows the average spot peer score.

You’ll notice that some bots (e.g., goodheart\_labs in 22nd, and slopcasting in 24th) have an average peer score that would have given them a rank in the top 5, except they didn’t forecast on enough questions (and consequently have much larger error bars). Of course, they may just have gotten lucky with their few forecasts, which is why we use the sum of peer scores to determine rankings: it incentivizes forecasting on more questions, which reduces our uncertainty (i.e., gives tighter error bars). # Are Bots Better than Human Pros? To compare the bots to Pros, we used only the 93 questions that were identical in the bot and Pro tournament and that were not annulled. We then aggregated forecasts of the “bot team” and the “Pro team” (see methodology section above). The bot team that was chosen is: * Panshul42 * pgodzinai * metac-o3+asknews * metac-o3-high+asknews * manticAI * TomL2bot * CumulativeBot * twsummerbot * metac-o4-mini-high+asknews * lightningrod We then calculated the head-to-head scores of the two teams, along with 95% confidence intervals using a t distribution. The average bot team head-to-head score was -20.03 with a 95% confidence interval of \[-28.63, -11.41] over 93 questions.

The negative score indicates that **the bot team had lower accuracy than the Pro team. A one-sided t-test demonstrates that pros significantly outperformed bots (p = 0.00001).** Below is an unweighted histogram of the head-to-head scores of the bot team.

The distribution of scores is generally symmetric with a small left lean and fat tails on both sides. Scores in Q2 were slightly more concentrated on 0, but generally fit the same shape as previous quarters. **Additionally, each individual Pro beat every individual bot.** We simulated a tournament where both pros and bots competed together as individuals on the 93 overlapping questions. Below is this leaderboard ordered by sum of spot peer scores. The top 19 participants are shown. Average spot peer scores are shown on the y-axis.

You’ll notice that if pros and bots competed together, all 10 pros would take 1st through 10th place (with metac-o3 getting 11th place). # Binary vs Numeric vs Multiple Choice Questions We also took a look at how bots did on each question type. When comparing the bot team and the Pro team on head-to-head scores, generally, bots did a lot worse at multiple-choice questions (score: -32.9) than at binary (score: -14.8) or numeric (score: -23.2). We also found that bots were worse at multiple-choice in Q1. Generally, the sample sizes in these comparisons are fairly small, so hold these conclusions lightly. Here are the average head-to-head scores for the 55 binary questions:

And here are the average head-to-head scores for the 17 multiple-choice questions:

And here are the average head-to-head scores for the 21 numeric questions (we do not include confidence intervals because our low sample size and a failed Shapiro test prevent us from validating the assumption of normality):

# Team Performance Over Quarters Now, let's compare how the bot team has done relative to the Pro team over the last 4 quarters. Below is a graph of Pro team vs bot team head-to-head scores over 4 quarters with 95% confidence intervals.

The values in the graph above are: * 2024 Q3: -11.3 \[-21.8, -0.7] * 2024 Q4: -8.9 \[-18.8, 1] * 2025 Q1: -17.7 \[-28.3, -7.0] * 2025 Q2: -20.03 \[-28.63, -11.41] Since the confidence intervals do not overlap with 0 (which would indicate equivalent performance), we can see that in most quarters the Pro team handily beat the bot team (except for Q4). However, we cannot discern any trend regarding whether bots are improving relative to pros due to the overlapping confidence intervals between quarters. For instance, the true spot peer score of the bots (if we ran on a lot more questions) might be -15 throughout all three quarters. Unfortunately, we just do not have enough questions to state the direction that bots are trending in (despite having enough sample size to show that pros are better than bots). One theory for the decrease in score from between Q3/Q4 and Q1/Q2 is the introduction of multiple-choice and numeric questions. As noted above, these were difficult for bots, and Q3 and Q4 only had binary questions. Another theory is that the Pro team has gotten better. They have had a while to practice as a team with each other, and practice improves performance. One other factor is that they have access to better AI tools that can speed up their own research process. Though the pros we asked said that learning to work together as a team, and borrowing each other’s forecasting patterns, were more helpful than AI tooling. Additionally, we have seen that the metac-gpt-4o bot started in 4th of 55 in Q3, 17th of 44 in Q4, 44th of 45 in Q1, and 82nd of 96 in Q2. Assuming that metac-gpt-4o can act as a control, this is evidence that bots have gotten better over the quarters. Though we have also seen metac-sonnet-3.5 go from 13th of 55 in Q3, to 42nd of 44 in Q4, to 34th of 45 in Q1, to 54th of 96 in Q2. This trend is much noisier. Some factors to consider: * Ranks are not a great way to measure this progress since they are sensitive to the number of new competitors that have joined late (since every new competitor starts at the middle of the leaderboard with 0 pts), and some competitors will drop early given early bad performance. * There have been slight changes in the underlying gpt-4o model over time, which would normally improve performance (though perhaps it’s hurt performance). * metac-gpt-4o saw slight prompt changes over the quarters. In Q3, it used perplexity for its news. In Q4, it used AskNews. In Q1, it had a minor prompt change for binary forecasts and started handling numerics/multiple-choice questions. Therefore, we believe that there is weak evidence that bots have gotten better on binary questions between quarters, and got worse in Q1/Q2 due to multiple choice questions. At the same time, pros have been improving, thus resulting in an increased gap between bots and pros. But generally (and statistically speaking), we can’t be sure of much about how bots have changed over time. # Bot Maker Survey At the end of Q2 2025, we sent a survey out to bot makers who won prizes. Winners were required to fill in this survey as part of receiving their prize. There were a total of 19 completed surveys from botmakers, with 7 of them ranking in the top 10 (there were 96 bots in the tournament). Below are some insights from this survey. ## Best practices of the best-performing bots To compare the bots on an apples-to-apples basis on forecast accuracy, we divided the total score by coverage to arrive at what we call a coverage-adjusted score. From a methodological standpoint, this was necessary because a few bots (e.g., goodheart\_labs and slopcasting) arrived late to the tournament but had very good scores in the time that they were present. The scores and official rankings of the botmakers are as follows (sorted by coverage-adjusted score):

We wanted to explore the difference between average coverage-adjusted scores on a couple of points. We chose some questions we were interested in testing the effectiveness of. Among the questions in the survey, we asked participants to check a box if they used code to “Take the median/mean/aggregate of multiple forecasts”, if they did “Significant testing via manual review of bot outputs (more than sanity checks)”, if they did “Testing via custom question creation/resolution”, and if they researched questions with “AskNews”. We then calculated the average score of those who said yes and no, took the difference, and found the upper and lower bounds using a 95% confidence interval. Please note that the sample sizes on these are small, and do not meet the n > 30 check for normality often required for meaningful bounds, so take these results with higher uncertainty.

The largest positive effects come from testing one’s bot via the creation and resolution of custom questions. Botmakers who did this had a higher coverage-adjusted score of 2,216 points on average, as compared to botmakers who did not engage in this practice. (95% confidence interval = +912 to +3,519). However, after talking with one of the participants who checked this box, we realized that there might be ambiguity about what “Testing via custom question creation/resolution” means. We were interested in whether any bot makers created their own questions that are not hosted on Metaculus. This would include creating, resolving, and scoring LLM-generated questions. We think at least one participant may have interpreted this as “testing on questions outside of the AI benchmarking tournament with a custom set-up, even if using pre-existing questions on Metaculus”. Though we also know at least one participant interpreted it as it was intended. We are unsure about the others. The second largest positive effect came from aggregation, which is taking the median or mean of multiple forecasts, rather than just one LLM forecast. Botmakers who did this had a higher coverage-adjusted score of 1,799 points (95% confidence interval = +1,017 to +2,582). The third highest positive effects came from doing significant testing via manual review of bot outputs (beyond sanity checks). Botmakers who did this had a higher coverage-adjusted score of 1,041 points (95% confidence interval = -223 to +2,305). In his advice to bot makers, our winner Panshul42 suggests that in development, one should “Measure performance not just by accuracy metrics, but also by how well the LLM reasons.” A lot of the advice that bot makers shared (see relevant section below) echoes the importance of this theme. One practice that did not make a demonstrable difference for botmakers was the use of AskNews. The AskNews result confirms what we found in earlier sections of this analysis regarding our in-house Metac Bots (i.e., that there was no clear “best search” option that stood out statistically between any of AskNews, Exa, Perplexity, etc). However, anecdotally, two of the most dominant bots, Panshul42 and pgodzinai, used AskNews for research. On the other hand, other high performers such as CumulativeBot, goodheart\_labs, and lightningrod did not. It is also notable that the top 3 botmakers were all either students or individual hobbyists, with #1-ranked Panshul42 putting in a total of 41-80 hrs (coverage-adjusted score of 6,256), #2 pgodzinai 16-40hrs (coverage-adjusted score of 4,837), and #3 CumulativeBot with 41-80hrs (coverage-adjusted score of 4,223). The next three participants were commercial entities: goodheart\_labs (coverage-adjusted score 4,107), manticAI (coverage-adjusted score 3,793), and lightningrod (coverage-adjusted score 3,667). Time spent on developing one’s bot had a fairly weak correlation of only 0.20 with coverage-adjusted tournament score. As part of the survey, botmakers selected a multiple-choice option for how many total hours (between all team members) have been put into developing their bot. We used the midpoints of time estimates and converted them into hours (e.g., 16-40 hours converted to 28 hours). We also converted 1 month to mean 160 work hours. We looked at time spent and its correlation with tournament score. Below are the responses we got (Note: one person responded with a custom “other” field) Forms response chart. Question title: What is your best estimate for how many total hours (between all team members) have been put into developing your bot?. Number of responses: 19 responses.

Forms response chart. Question title: What is your best estimate for how many total hours (between all team members) have been put into developing your bot?. Number of responses: 19 responses.

LLM calls per question had a correlation of 0.40 with a bot’s coverage-adjusted score, which is typically considered weak to moderate. This makes intuitive sense that more LLM calls per question would help due to it providing the model with more chances to research, engage in Fermi-style reasoning, aggregate multiple models, etc. However, it may be that the real reason for the mild correlation is that more calls are needed in order to properly aggregate forecasts, which we found to create some of the largest positive effects.

Q2 AI Benchmark Results: Pros Maintain Clear Lead

Another Year of AI Benchmarking: The Plan

Q1 AI Benchmark Results: Pro Forecasters Crush Bots

Analysis of Automated Prompt Engineering for Forecasting

Build Your Own Forecasting Bot in 30 Minutes With Our Tutorial Video

Q4 AI Benchmarking: Bots Are Closing The Gap