Attention
The woods are lovely, dark and deep
Keys K
Queries Q
lovely
dark
deep
woods
lovely
dark
deep
woods
×
Values V
lovely
[
+0.60 -0.20 +0.15 +0.42
]
dark
[
-0.15 +0.45 +0.28 -0.08
]
deep
[
+0.25 +0.30 -0.12 +0.55
]
woods
[
+0.10 -0.35 +0.48 +0.22
]
+=
Updated "woods" embedding
[
+0.22 +0.18 +0.12 +0.27
]
Fig. 5. Woods looks up its keys: the strongest Q·K dots land on lovely, dark, and deep. Softmax turns those scores into weights on each V; the mix is the woods embedding.