Over the last year, Metaculus has run a \$120k tournament split over 4 quarters where bots compete to forecast future events. There have been many interesting results so far! You can find them here: [<u>2024 Q3</u>](https://www.metaculus.com/notebooks/28784/aibq3results/), [<u>2024 Q4</u>](https://www.metaculus.com/notebooks/35291/q4-ai-benchmarking-results/), and [<u>2025 Q1</u>](https://www.metaculus.com/notebooks/38673/q1-ai-benchmarking-results/) (the 2025 Q2 analysis is in progress).

We are excited to announce that we are renewing the AI Benchmarking Tournament for another year with \$175k in prizes! This new iteration brings changes to the tournament’s structure. The tournament will now be split into 2 series:

* **\$1k Bi-weekly MiniBench**: Minibench will be a series of back-to-back 2-week-long \$1k tournaments of \~60 questions each. Questions will at first be automatically created and resolved from curated data sources (e.g. FRED, Google Trends, etc.), and will eventually include LLM-generated questions. The purpose of this tournament is to provide fast feedback loops for participants to test the quality of their bots and to lower the barrier to entry for new participants. Additionally, it will help highlight the best forecasting LLMs faster than once a quarter. Due to automation, we expect this tournament to be slightly noisier, but it should provide an interesting point of reference. The first MiniBench will start sometime in August once we have the automated pipeline created.
* **\$50k Fall/Spring/Summer AIB**: The main AI Benchmarking Tournament (AIB) will now happen 3 times a year, and will restart in September to align with the Metaculus Cup. Each season will have a \$50k prize pool, and around 300-500 questions. Most (if not all) questions will be imported from the Metaculus Cup and the main site to compare bot performance against the community prediction.

We encourage people to prepare early for these tournaments. Here are some ways you can continue participating until September.

* **\$7k Market Pulse Tournament**: Bots are eligible for prizes in the [<u>Q3 Market Pulse </u>](https://www.metaculus.com/tournament/market-pulse-25q3/)tournament. Bots will need to handle numeric group questions, and continuously update forecasts during the question lifetime (unlike the previous AIB tournaments).&nbsp;Our template bot now handles group questions.
* **Metaculus Cup**: The Metaculus Cup is a great way to test your bot and compare yourself against human participants. This is the most popular human tournament on Metaculus, and though bots are not eligible for prizes, it can help you measure the strength of your bot on diverse questions.
* **Ask for Credits and experiment**: Metaculus has sponsored the LLM costs of participants in past AIB tournaments. If you want to run experiments or need credits for the above tournaments, please send us an estimate for how much money you think you can use productively (reach out to ben \[at] metaculus \[.com]). If you want 1-day feedback loops, you can use the [forecasting-tools](https://github.com/Metaculus/forecasting-tools?tab=readme-ov-file#benchmarking) Python package to score your bot against questions on the site by making the assumption that the community prediction is the "correct" prediction.

We will be sharing more information about the Fall AIB Tournament soon, but until then, we encourage you to use the resources from AIB Q2 to set up your bot. The AIB home page (which currently includes the Q2 tournament links and a 30min set-up-a-bot tutorial) can be found [<u>here</u>](https://www.metaculus.com/aib/). If you have any questions, please feel free to comment here, on our Discord, or reach out to ben \[at] metaculus \[.com]!