Key, Query, and Value are vector representations used in the Attention Mechanism to calculate how much focus the model should place on other parts of the input.
Key Concepts
- Analogy: They are often compared to retrieval systems where a Query is matched against Keys to retrieve Values.
- Calculation: These vectors are derived from the Vector Embedding of the input tokens.
- Attention Score: The “Attention Score” is computed using these vectors to determine the relevance of each token to the others.
