Here is the full list of advice given by bot makers that completed the [2025 Fall bot survey](https://www.metaculus.com/notebooks/43337/fall-2025-futureeval-survey/). We share this so that future bot makers can build upon past work and benefit from learning from the mistakes and failures of others.

## AI Summary of Advice

These are the qualitative patterns that AI found in the quotes below

### 1. Research quality is the #1 differentiator — and it should be agentic, not one-shot

The most repeated lesson: forecasting quality is bottlenecked by the information reaching the final LLM call, not the reasoning itself. Bot makers consistently recommended **iterative/agentic search** (generate queries → search → identify gaps → search again) over single-pass research, and emphasized that **external search is mandatory** due to training cutoffs and hallucination. Native web search tools from OpenAI/Anthropic/Google were praised as more reliable than custom pipelines. Keeping the context window clean (avoiding irrelevant info) was also repeatedly called out.

### 2. Use frontier reasoning models, but don't over-engineer the reasoning layer

Modern reasoning models (GPT-5.1, Grok-4.1, Claude, etc.) are strong enough that **clear prompting beats clever architecture**. Multiple makers said their attempts to "improve" the template made things worse, and that default models with good prompts go surprisingly far. Reasoning models vastly outperform non-reasoning ones. Code execution capability was flagged as important for questions requiring calculation.

### 3. Ensemble, aggregate, and reduce variance

Running **multiple forecasts (5+) and taking a median/aggregate** was broadly endorsed, as was ensembling across different LLMs. Several makers warned that cutting forecast counts to save cost backfired badly due to variance. Related techniques mentioned: debate, multiple personas, Monte Carlo, Delphi methods — though one maker warned that naïvely feeding personas into a final merge step underperforms.

### 4. Test rigorously, read the logs, and sense-check outputs

Bot makers repeatedly regretted **not testing systematically** and not watching their bot's actual behavior. Specific lessons: use **minibench** for iteration (since resolved-question backtests leak), run minimal-change experiments, sense-check weird predictions, and catch technical bugs early — one maker lost \~\$500 in prize money from a single uncaught bug affecting a rare question format. Past-casting on domains where you're personally an expert surfaces reasoning errors quickly.

### 5. Handle question-type edge cases and avoid known pitfalls

Recurring concrete gotchas:

* Bots often mistake **unresolved questions as already resolved**, producing catastrophic 99% forecasts — add explicit guards.
* **Continuous/numeric questions** need special handling; the template's approach is naive. One maker had success predicting mean + standard deviation rather than percentiles.
* **Clipping/extremization** of final predictions is underused.
* Include units, assumed start dates, and clear output-format instructions in prompts.
* Make sure your bot reliably submits on *every* question within the submission window.

## Open Source Bots

We asked if bot makers were willing to open source their bots. The below bot makers agreed. You can connect bot makers to their performance via the [Fall 2025](https://www.metaculus.com/tournament/fall-aib-2025/) and [Spring 2026](https://www.metaculus.com/tournament/spring-aib-2026/) leaderboards. Here's the bulleted list of bot makers and their repo links:

* **nostreambot (best open source bot at 9th)** — [https://github.com/No-Stream/metaculus-bot](https://github.com/No-Stream/metaculus-bot)
* **geemus+bot (14th)** — [https://github.com/geemus/metaculus-ai-forecasting](https://github.com/geemus/metaculus-ai-forecasting) (end of fall-aib state: [https://github.com/geemus/metaculus-ai-forecasting/commit/646c5cc2a8c2b131ce63256494eafc72336e25a1](https://github.com/geemus/metaculus-ai-forecasting/commit/646c5cc2a8c2b131ce63256494eafc72336e25a1))
* **OliveTestBot (16th)** — [https://github.com/theolivecode/metac-bot-template/tree/main](https://github.com/theolivecode/metac-bot-template/tree/main)
* **BottyMcBotFace (26th)** — [https://github.com/jlaudon/metac-bot-template](https://github.com/jlaudon/metac-bot-template) (repo is public, bot runs from HEAD)
* **alexwcohen\_bot (27th)** — [https://github.com/alexwcohen/metac-bot/blob/main/main.py](https://github.com/alexwcohen/metac-bot/blob/main/main.py)
* **InstitutPelFutur (30th)** — [https://github.com/altrome/metac-bot-template](https://github.com/altrome/metac-bot-template)
* **Yrambot (34th)** — [https://github.com/omagu-byte/metac-bot-template--YRAMBOT](https://github.com/omagu-byte/metac-bot-template--YRAMBOT)
* **gontxal0\_bot (56th)** — [https://github.com/herman-aukera/metac-bot-ha](https://github.com/herman-aukera/metac-bot-ha)
* **samcodes (65th)** — [https://github.com/samuelukwu27-star/forecaster](https://github.com/samuelukwu27-star/forecaster)

You can find a larger list of open source bots on our [resources page](https://www.metaculus.com/notebooks/38928/ai-benchmark-resources/#open-source-bots).

## Quotes

Below is the advice that our bot makers shared when asked at the end of the survey "In summary, what should other bot makers learn from your experience?"

***

"Be weary of evaluating your forecasting system on resolved questions. I tried out many personality, llm-as-a-judge checks, web scraping techniques, to find data leaks and so it is hard to judge the quality of your system and iterate without access to the internet for answering questions. So take advantage of the minibench as a way to iterate properly over your system."

***

"I'm not totally sure what made the biggest difference, since I didn't do a ton of systematic testing. My best guess is the most important pieces were pretty mundane:

1\. Using the most powerful reasoning model available (that ended up being GPT 5.1 when I was forecasting). I think this probably made a difference, just based on benchmarking studies showing bigger models tend to do better broadly.

2\. Making sure my bot could search recent news (I couldn't get AskNews to work so just used GPT 5.1:online). I realized some of my early forecasts were way off because I wasn't actually searching recent news. I also initially wasn't taking advantage of forecasts from Metaculus or other websites on the same or similar questions.

I spent a decent amount of time making sure my bot followed good, Tetlock-style forecasting practices (base rates, inside/outside view, consider the opposite, etc.). I'm not sure how much of a difference that made but guessing it was also helpful.

If I were doing this again, I'd use Claude Code to do more sophisticated things (e.g., multiple personas, ensemble models) and do more testing to see what was working and what wasn't/calibrating based on more training data."

***

"1. Multiple personas, cheaply run, going straight into the output, was a mistake. The final stage is bad at being genuinely critical

2\. Make it quicker. I missed several weeks at the start.

3\. grok-4.1-fast is amazing

4\. qwen-3-235b-instruct is great for its price

5\. Using LLMs to combine distributions was silly and kept driving me insane and I should've moved away from that earlier

6\. Read the logs. So many subtle issues that took me far too long to notice.

7\. Agents are overrated. I tried them, and ended up going back to traditional pipelines. Maybe I was doing it wrong, the atomic-agents library seemed pretty poorly designed

8\. openai's mini/nano models are solid

My system consisted of a research system with lots of different parallel submodules, including my own agentic researcher, that then went into a report-writing system, that then went into four prediction systems:

1\. Monte carlo sim

2\. Generate perspectives \=> Critique \=> Output

3\. Generate with a bunch of personas (""systems thinker"", ""historian"", etc)

4\. Work out a base rate \=> update with new data + prediction

Then ""stage 1"" critiquers, which take those 3 systems and produce an output

Then ""stage 2"" which merges the stage 1s - this is an LLM call!

I learnt that an agentic researcher with a bunch of tools just doesn't work very well compared to a bunch of dedicated pipelines, though maybe I just shouldn't have used the atomic-agents library!

I also spent the tournament slowly moving away from ad-hoc prompts to XML-styled ones. I don't know that it improved performance, but it certainly made my life easier."

***

"I found running experiments with minimal changes to be extremely helpful, especially because the things I assumed would have a large impact on performance would often generate essentially identical forecasts after making changes. I wrote about some of these experiments at predictably.substack.com.

Other things such as regularly sense-checking forecasts and keeping an eye on weird predictions are also very useful for catching errors.

Technical errors could be extremely costly. I made a change that impacted a specific relatively rare question format, which I didn't catch immediately, and this change ended up costing me over 1000 points from only 3 questions. In terms of prize money this was a roughly \$500 mistake."

***

Having a concrete use case really pushed me to better understand how to structure and iterate on agentic workflows. I learned a lot about LLMs generally, as well as stumbling a few times. It's easy to continually add things, but takes some work to pair them back and refine them (but this is very worth doing as well). I found good success with the cheaper models through careful prompting and using delphi methods to round out things. But I've also seen bigger improvements in the spring taking that and giving it nicer models as well. That said, I'm still seeing a good deal of variability, so it's hard to know when luck/coincidence vs real changes have taken place and it's good that minibenches keep providing us with some additional feedback to help us keep iterating. I also learned a lot from writing a lot of the initial code myself, but have also shifted more and more into agentic coding (which has it's own pros/cons). I think both have their place.

***

It is very easy and takes little effort to perform better than the template/metac\_model bots and is a great opportunity to practice programming and gain experience working with LLMs.

***

Reasoning models vastly outperform no reasoning. It is important to give your bot instructions or have checks so that your bot does not think a question has already resolved when it has not, leading it to predict 99% on a question with a low probability of positive resolution (this is a good way to get a massive negative score). Using a single call with web search is cheaper than doing a separate research call and forecasting call.

***

Good research is crucial. Debate is very powerful. After the first month, I cut number of forecasts to 1 which really hurt my average score on binary/multiple choice due to the variance bleeding edge. Either use more forecasts and aggregate or reduce variance (gpt-5 was especially bad for this because you couldn't adjust temp)

***

My prompting was good. If claude code was better in the fall my intermediate functions would not have been as sketchy because my python skills are horrific and I would have done better, but I think these barriers are nonexistent now and the only thing that matters is the ability to think up a smart prompting sequence. I think the important thing to understand is that LLMs will just spit out the information you give them and do what you tell them to and they have zero judgement, so you have to exploit information asymmetries in order to create superficial judgement so that it has some ability to do more than just aggregate the information it's given (as then it is just a news reaction function with time decay rather than an actual forecasting bot). 

***

I'm not a dev and going forward I will be doing a lot more testing and making much more use of minibench. But I would say simply use the best available models and focus on improving the research input to your forecaster. 

***

Ensembling across LLMs is worth doing. Continuous question handling in the template bot is naive and can be improved upon.

***

External research is necessary for bots to forecast no matter how good the bot's research pipeline is. This is due to training data cutoff dates and tendency of hallucinating. A mix of research methods depending on the question type are also beneficial in the long run. For my bot in the fall, the bot reliance on its own research capabilities worked for certain questions and failed woefully for other type of questions particularly numeric question forecasts.  I noticed for certain questions where it performed better than most bots, this was due to its cut off date being 2024. Bots should add a layer of extremization, ensure to have robust research and reasoning.  

***

1\) The quality of the competition is quickly improving. Probably best shown by number of bots outperforming both the community and the best base metac bots. 2) The Metaculus framework is good, but to be in the top third requires going beyond it, or at least modifying it.

***

Bot makers can build on the existing metaculus template. Ensure to add a solid structure and understand how llms forecasts compared to humans.

***

The Metaculus template bot is a good way to learn how to make a bot. One should just follow the guidelines.

***

The top reasoning models have gotten very good. You don't need to reinvent reasoning, but you do need to do a good job of explaining very clearly what you want these reasoning models to do. Getting accurate, relevant, and unbiased information to the model you're using to generate forecasts is a big differentiator. Finally, there is only so much that can be accomplished purely through reasoning. Many questions require an amount of calculation that can only reasonably be done by writing code, so allowing some sort of code execution makes a big difference.

***

"Key aspects that resulted in good forecasts:

\- Used multiple replicates to capture uncertainty before synthesizing into final forecast

\- Avoided polluting the context window with irrelevant or partially relevant information

\- Included question details such as unit for numerical questions, along with detailed guidelines on how to generate a well-calibrated forecast

Key aspects that resulted in bad forecasts:

\- Missing diversity of research

\- Insufficient prompt details such as the assumed start date for ""Will X occur by"" questions"

***

Prioritise research and context hygiene. Having reliable code that forecasts on every question is very important. Keep track of token costs and ensure you can use a frontier model for final forecast whilst also taking a statistical median of 5+ runs.

***

The default foundation models generally tend to get better and better over time with little tweaking. I switched this tournament from using Perplexity to using the default web search capability tools and didn't see any noticeable difference in results. Although I've experimented with calibrating the bot's predictions, it's hard to say if this is actually better or not since the models are generally improving and the calibrations might be less and less necessary. While there are many different strategies and approaches one could use for these tournaments to get slightly better performance, I think, like almost every other measure with LLMs, get pretty far by just using the default LLM performance with decent prompts.

***

The template bot is better than I expected. I tried to make some low-effort adjustments to it to improve it, but I believe I made it worse, rather than better.

***

"I believe the bot currently needs some guidance, and it's especially useful to have instructions on how to predict different types of questions.

It's important to constantly check if an issue arose that prevents the bot to predict, as there is a short time window to submit the forecasts."

***

The community continues to underrate the benefit of clipping their predictions.

***

I think ensuring the final llm prediction call has enough information to make a decent prediction is essential. No matter what model you use, if it doesn't know enough about the current state of the world, it's going to have a hard time. So I think a form of dynamic search (like agentic search, or some form of being able to investigate further) is important. For example in additional to AskNews (which was already decent), my loop involved generating queries, searching, checking results, and generating new queries to address remaining information gaps. I also think what you request of the llm is important. For example I think my bot did very well on numeric questions because I requested that it focus on producing a mean and standard deviation instead of a bunch of percentiles which allowed it to focus solely on the single most likely value and the uncertainty (this assumed normal distributions and neglected the tails which isn't good, but I think it allowed it to focus on predicting the general outcome more easily). I also think it's promising to think a bit more about how to lean into the strengths of the latest models, like their agentic and coding abilities.

***

I'm not convinced that one-shot research steps can be competitive with the more agentic/adaptive search based approaches. Humans look things up, find more questions to ask, and then look more things up. Bots should behave the same way.  

***

I was experimenting with trying to make my bot as efficient as I could, with the experience of the fallbot and further experiences in the spring I've decided that this is more of a wrong direction.

***

Small models can rival or beat large flagship models if they are fine-tuned via RL with the future as the label. 

***

Native web search tooling (openai/anthropic/google) is very useful (agents are trained to use them effectively during reasoning using RL + the actual retrieval is more reliable)

***

Whatever your idea is I think it’s best to either past cast or have the bot predict on questions where you’re an expert. I made sport prediction questions and caught a lot of reasoning errors that way. 

***

Get predictions from many different LLMs to use in creating your final prediction.