The attention mechanism in transformer models, which calculates correlations between pairs of tokens, takes quadratic time in the input size, making it a time bottleneck for transformer operations.

Attention mechanisms have incurred 7 years' gradual innovations, showing that the transformer architecture is still evolving and improving.

In March 2023, a new state-of-the-art model was being released almost every other day, indicating rapid advancements and competition in NLP models.

Currently, PaLM and Megatron NLG 530B by Nvidia are leading the chart on performance, demonstrating the dominance of transformer models in NLP benchmarks.

Despite advancements, some models like the Eagle 7B have surpassed traditional transformers in evaluation benchmarks, suggesting potential competition to transformer dominance.