# Overview
Here is a quick update on work done for the Foresight Grant as of Oct 29. Thus far, the general goal has been to 1) create a powerful open source forecasting bot, and 2) as a byproduct, create useful research tools for humans. I have had time to run 5 experiments. I have notes to create a deeper write-up to dive into these in more detail, but here are the preliminary highlights:
| Grant Deliverable | Grant Description | Status |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Open Source Code | "An open-source GitHub repository will be made so others can take full advantage of the development work done." | ✅ Completed ([link](https://github.com/CodexVeritas/forecasting-tools)) |
| Demo site | "A demo site with the ability to interact and use successful tools, and the current state of the bot" | ✅ Completed ([link](https://forecasting-tools.streamlit.app/chat)) |
| Python Package | "A Python package (allowing integration into other people’s code in seconds)" | ✅ Completed ([link](https://pypi.org/project/forecasting-tools/)) |
| Accurate forecasting bot | "A bot that is better than a human (at a bare minimum the average American citizen, but shooting for the top 5-25% of forecasters generally)" | ✅ Completed (better than the public, probably in top 15-50% of forecasters based on [Metaculus Cup](https://www.metaculus.com/tournament/metaculus-cup-summer-2025/?page=3), depending on how you adjust for participation) |
| Overview of experiments | "A document covering the success and failure of each experiment in improving the accuracy of the forecasting bot" | 🚧 Partial. This document counts. One iteration has been written up in detail [here](https://www.metaculus.com/notebooks/38421/results-of-automated-prompt-engineer-for-forecasting/). Some still need to be published. |
| 6 Months of work | "There are 25 weeks in a 6-month period, so a total of \~12 iterations." | 🚧 Partial. 2 months of work with 4.5 iterations. |
| Human Usable Tooling | "I plan to create an open-source forecasting bot powered by a series of human-usable AI-powered research tools to help the forecasting ecosystem make more forecasts and more accurate forecasts" | ✅ Completed (though more can be made - see demo site, and especially question generator) |
# Experiments
1. **Question Decomposition tool**: The first experiment was to create a first-pass AI Question Generator and see how this improves forecasting. The tool didn't have much of an effect on increasing forecasting accuracy, but has been useful to humans. Some team members have been using it regularly, and I believe Foresight has used it as part of preparing for the[ <u>AI Pathways Tournament</u>](https://www.metaculus.com/tournament/foresight-ai-pathways/). The demo of the tool can be found by going to this[ <u>chatbot</u>](https://forecasting-tools.streamlit.app/chat) and clicking "Brainstorm Forecastable Questions" (or by giving a topic of interest and asking the chat to use the "Decompose into questions" tool to make questions).
2. **Prompt optimizer**: I realized that there were a lot of possible prompts in the previous iteration which might have worked better, but didn't have time to try. So my next goal was to improve iteration speed by allowing AI to automatically generate prompt ideas and test how well they work in a bot. This brought up some interesting results which I decided to publish[ <u>here</u>](https://www.metaculus.com/notebooks/38421/results-of-automated-prompt-engineer-for-forecasting/). Generally, it seems that the optimization framework had statistically significant improvements for weak models like GPT-4.1-nano, but had more moderate improvements for GPT-4.1.
3. **Dataset Analyzer**: In the next iteration, I wanted to build some tooling I could use in future iterations, while also seeing if I could create a tool that could proactively find, download, and forecast from datasets. Generally, I got computer use and data analyzer tools to work together, but the computer use model was inconsistent enough that it wasn’t worth testing its improvement in forecasting.
4. **Research optimizer**: For the next iteration, I was curious whether I could get AI to not just optimize prompts, but optimize research processes (e.g. what research tools and queries to call in which order, etc). Though this had similar effects to the prompt optimization, it did not create noticeably better prompts. Additionally, I tried to optimize o3 (the best model at the time), and found no improvement in forecasting ability. This is making me lean towards the belief that 1) prompt engineering is only really useful for lower quality models, and 2) for whatever reason, models don’t benefit from control over their research process (at least at this point in time).
5. **Testing optimized prompts in real tournaments**: Thus far I have evaluated bots based on how well they adhere to the site’s community prediction. I wanted to test this in the real world, and so have recently been running our best optimized prompt (for GPT-4.1) and have found positive results so far. In our[ <u>MiniBench</u>](https://www.metaculus.com/tournament/minibench/) series (i.e. short, fast, but lower-quality questions) this bot has beat the original control bot consistently by 10-30 places on the leaderboard of 70 bots (except for one tournament where it only beat it by 2 places). This bot is also currently in 10th in the[ <u>main tournament</u>](https://www.metaculus.com/tournament/fall-aib-2025/) (though this is probably mostly noise since very few questions have resolved so far).
In addition to this work, I have also been running the AI Forecasting Benchmark Tournament. Part of our research from this program has found that competitors spending weeks and even months developing bots have not been able to beat a simple one-shot bot running a frontier model. This trend wasn’t super clear when the grant started, but seems fairly consistent over the last year. You can see some of this in my most recent[ <u>analysis of the tournament</u>](https://www.metaculus.com/notebooks/40456/q2-ai-benchmark-results/). This seems to echo the trend I found, where prompt optimization has diminishing returns as models get stronger. I’m hoping to run an optimization run of GPT 5 and Sonnet 4.5 to see if the trend continues. Though, if this is the case, all that needs to be done to have the “best forecasting bot” is to wait for a frontier AI Lab to release a new model. These bots are open source, and are fairly cheap, easy to use, and “ok enough” at forecasting to start being used in low-stakes decisions. They are better than the public and though less tested, are better than many forecasters (as seen in Summer Metaculus Cup)
# Commentary
Given the decent performance of bots at this time, and the diminishing returns of further efforts, at this point, question generation seems more promising as a path forward for grant activities.
Good question generation can be used in a number of different ways:
1. **Fully automated forecasting tournaments**: Reap the value of starting a tournament on Metaculus without the overhead of manual writing. Bots can forecast these without needing to put up a large prize pool. This would mean Foresight (or any other org) could create multiple[ <u>AI Pathways Tournament</u>](https://www.metaculus.com/tournament/foresight-ai-pathways/)s in the time it takes normally takes to create one.
2. **Worldbuilding and grounded narratives**: Question generation can lead into other applications like worldbuilding. By asking a lot of questions, you can weave a more coherent view of the future, and assign probabilities to each part of the narrative.
3. **Causal Analysis**: By decomposing questions into subquestions and showing these in graphical format, it becomes much easier to find root causes, neglected uncertainties, actionable solutions, and other relevant items for solving problems.