Introducing Open Review
26
_From Sylvain, your friendly neighborhood Product Manager._ I started my journey at Metaculus as a volunteer Moderator, helping question writers elevate their submissions up to the Metaculus standard. [Some](https://www.metaculus.com/questions/15627/russian-nukes-in-belarus-before-2024/) [of](https://www.metaculus.com/questions/5356/2020-us-election-concession/) [our](https://www.metaculus.com/questions/2671/us-astronauts-land-on-moon-by-2025/) [best](https://www.metaculus.com/questions/4808/will-the-median-prediction-on-this-question-fall-in-range-1-48-or-95-99/) [questions](https://www.metaculus.com/questions/9459/russian-troops-in-kiev-in-2022/) were written by regular Metaculus users. Still, I know our question approval process hasn't been perfect. We want to make it faster, more transparent, and more fun, without compromising question quality. That's why I'm very excited about Open Review. Starting today, the queue of user-submitted questions is public. You can find it [here](https://www.metaculus.com/questions/?order_by=-votes&status=pending), or by using the “In Review” link in the main feed sidebar. <img src="https://metaculus-media.s3.amazonaws.com/open-review.gif" /> This is very much an experiment, so expect some course corrections as we learn more. Here’s the process we have in mind: 1. As before, when a question author is satisfied with their draft, they submit it for review. 2. The question appears in the Review queue. Other users read it and leave comments and suggestions for improvement. The author answers comments and improves their question. 3. Authors who are so inclined can even invite others to be coauthors, so they can directly collaborate on improving the question. 4. Users upvote questions that are both interesting and ready to go. 5. Moderators still approve questions, but now they prioritize highly upvoted questions. This last point was much discussed internally: should only moderators be able to approve questions? Ultimately, we decided to keep a moderator in the loop. Question quality is too important to the platform. Now, this doesn't mean we'll never open up the approval process. This experiment will shed light on a few points—including whether a distributed review process produces Metaculus-quality questions. After we collect more information, we’ll reassess and share our conclusions and next steps. Open Review is a big experiment for us and potentially a big opportunity for Metaculus to open more questions more quickly. If you have thoughts or feedback, we want to know.