Your submission is now a Draft.

Once it's ready, please submit your draft for review by our team of Community Moderators. Thank you!

You have been invited to co-author this question.

When it is ready, the author will submit it for review by Community Moderators. Thanks for helping!


This question now needs to be reviewed by Community Moderators.

We have high standards for question quality. We also favor questions on our core topic areas or that we otherwise judge valuable. We may not publish questions that are not a good fit.

If your question has not received attention within a week, or is otherwise pressing, you may request review by tagging @moderators in a comment.

You have been invited to co-author this question.

It now needs to be approved by Community Moderators. Thanks for helping!


{{qctrl.question.predictionCount() | abbrNumber}} predictions
{{"myPredictionLabel" | translate}}:  
{{ qctrl.question.resolutionString() }}
{{qctrl.question.predictionCount() | abbrNumber}} predictions
My score: {{qctrl.question.player_log_score | logScorePrecision}}
Created by: isinlor and
co-authors , {{coauthor.username}}
AI Training and Compute

Make a Prediction


A statistical language model is a probability distribution over sequences of words. Due to Google and OpenAI work big pre-trained language models gained recognition as a multitask and few-shot learners bringing as a step closer to general artificial intelligence.

Big pre-trained language models contain a lot of implicit knowledge about the world, however retrieval of that knowledge is not always reliable. These models are also expensive to update with new knowledge, because to do so they would require additional training.

One way to address above issue could be augmenting language models with the capability of traditional search engines like Google. An example attempt at this task is the paper REALM: Retrieval-Augmented Language Model Pre-Training utilizing relatively small 330M parameters model.