Your submission is now in Draft mode.

Once it's ready, please submit your draft for review by our team of Community Moderators. Thank you!

Submit Essay

Once you submit your essay, you can no longer edit it.

Pending

This content now needs to be approved by community moderators.

Submitted

This essay was submitted and is waiting for review.

Will a language model with at least 100B parameters trained to do external information retrieval exist before 2023?

Question

A statistical language model is a probability distribution over sequences of words. Due to Google and OpenAI work big pre-trained language models gained recognition as a multitask and few-shot learners bringing as a step closer to general artificial intelligence.

Big pre-trained language models contain a lot of implicit knowledge about the world, however retrieval of that knowledge is not always reliable. These models are also expensive to update with new knowledge, because to do so they would require additional training.

One way to address above issue could be augmenting language models with the capability of traditional search engines like Google. An example attempt at this task is the paper REALM: Retrieval-Augmented Language Model Pre-Training utilizing relatively small 330M parameters model.

This question asks will a language model with at least 100B parameters trained to do external information retrieval exist before 2023?

This question will resolve positive when a language model with at least 100B parameters trained to do external information retrieval will be announced and negative if no such model will be publicly known to exists before 2023.

Importantly, the model must have at least 100B parameters and it must be trained by some means to do external information retrieval as in the REALM paper mentioned above. Just augmenting 100B model with e.g. TF-IDF after per-training will not suffice. The model must be aware of the external information retrieval during the training procedure. The specifics of achieving that goal are not relevant, so any method applied during training will suffice.

Make a Prediction

Prediction

Note: this question resolved before its original close time. All of your predictions came after the resolution, so you did not gain (or lose) any points for it.

Note: this question resolved before its original close time. You earned points up until the question resolution, but not afterwards.

Current points depend on your prediction, the community's prediction, and the result. Your total earned points are averaged over the lifetime of the question, so predict early to get as many points as possible! See the FAQ.