Transformer
The neural network design behind nearly every modern language model.
The transformer, introduced in 2017, replaced recurrence with attention: instead of processing a sequence step by step, every token looks at every other token directly and decides how much to weight each one. That parallelism made training on huge datasets possible, which is what unlocked the scaling that produced today's models.
The cost of that choice shows up later. Attention is quadratic in sequence length, so long contexts get expensive, and because every token attends to every other, distant tokens compete for a finite amount of influence. That competition is the mechanism behind attention degradation, and it is why long conversations get wobbly.
Related terms
Attention
The mechanism that lets each token weigh the relevance of every other token.
LLM
A model trained on a huge amount of text to predict the next token. The engine behind most AI products.
Context Window
Everything the model sees on one request. Finite, and it decides what fits.
Neural Network
A stack of weighted layers that transforms input into output by adjusting numbers.
In-context Learning
The model adapts to a task from the context alone, without any training.
Want this working inside your team?
It starts with one conversation about your workflow.