Attention Relationship
Hubungan Atensi
The pairing between two tokens. Meaningful pairs influence each other more than unrelated ones.
An attention relationship is the pairing between two tokens. There is one for every ordered pair in the context, so a context of N tokens has about N squared of them, and the model evaluates all of them on every forward pass.
The relationships are not equal. Some pairs are meaningful: the subject and its verb, a variable and the type it is declared as, the question and the passage that answers it. Those pairs get more influence than unrelated neighbours. A context that contains the right meaningful pairs produces better work than a larger context full of text that never relates to anything else in it.
This is the practical reason chunking matters in retrieval. Cutting a document at the wrong place breaks the pairs that made it understandable, and a chunk that lost its relationships is a chunk the model has to re-derive from scratch. It is also why repeating something, even briefly, in two places can work better than saying it once in the right place: the repetition creates a second meaningful pair.
Related terms
Attention
The mechanism that lets each token weigh the relevance of every other token.
Attention Budget
Each token has a finite amount of influence to distribute across the rest of the context.
Attention Degradation
Each token has a limited influence to spend, and later tokens crowd out earlier ones.
Retrieval
The step that finds the material a model will answer from.
Embedding
A list of numbers that places text near other text with similar meaning.
Want this working inside your team?
It starts with one conversation about your workflow.