A statistical language model is a probability distribution over sequences of words. Due to Google and OpenAI work big pre-trained language models gained recognition as a multitask and few-shot learners bringing as a step closer to general artificial intelligence.
Big pre-trained language models contain a lot of implicit knowledge about the world, however retrieval of that knowledge is not always reliable. These models are also expensive to update with new knowledge, because to do so they would require additional training.
One way to address above issue could be augmenting language models with the capability of traditional search engines like Google. An example attempt at this task is the paper REALM: Retrieval-Augmented Language Model Pre-Training utilizing relatively small 330M parameters model.