Question
What will be the best score on the SWE-Bench (unassisted) benchmark before 2025?
Resolved:29.38Total Forecasters25
Community Prediction
30.9
(29.4 - 34.8)
Make a Prediction
CDF
| Quartiles | community | My Prediction |
| lower 25% | 29.44 | — |
| median | 30.87 | — |
| upper 75% | 34.77 | — |
What was the final result?29.38
Community Peer Score
19.6
Community Baseline Score
59.1
Authors:
Opened:Mar 24, 2024
Closes:Dec 31, 2024
Resolves:Jan 1, 2025
Spot Scoring Time:Mar 27, 2024
What will be the best score by an AI on the full Humanity's Last Exam (HLE) before 2026?
59.4
When will an AI achieve an score of 1.5 or higher in the RE-bench at any time budget between 8h and 32h?
20 Aug 2027
What will be the best non-human SAT-style score on the hard subset of the QuALITY dataset by January 1, 2030?
97.1
Authors:
Opened:Mar 24, 2024
Closes:Dec 31, 2024
Resolves:Jan 1, 2025
Spot Scoring Time:Mar 27, 2024
What will be the best score by an AI on the full Humanity's Last Exam (HLE) before 2026?
59.4
When will an AI achieve an score of 1.5 or higher in the RE-bench at any time budget between 8h and 32h?
20 Aug 2027
What will be the best non-human SAT-style score on the hard subset of the QuALITY dataset by January 1, 2030?
97.1