attention mechanism , a cool perspective xd

05 Feb, 2025

hey all, i hope you are doing well , it's a sunny day with a cool breeze flowing in here in jodhpur with no lectures to attend today , perfect day to take a cup of coffee and your laptop , sit in your lab and write some cheeky ml stuff.

today we are going to discuss in detail about something very fundamental but very important called self attention. i had covered attention mechanism in detail in one of my previous blogs but I do want to cover self attention again. i mean i am thinking to umm.. kind of structure my blogs in a certain way such that we start from attention again and then start heading towards transformers and then we can discuss some foundation generative models from absolute scratch :) .... sounds cool no?

sooo , let's begin.....

what is attention ?

wait , you have been reading and reached till here , you would have focused on some words more than others right ? yeahhh , you would be like , yes bro , i paid more "attention" to some words like 'sunny day' , 'cheeky' , 'jodhpur' , 'no lectures' ...... , i feel all was okay till you reached what is attention , that's the sad part :( ?? okay am sorry for that part ... just bear with me a lil more on bearblog.

i assume you all hate math , no ? you hate? you love ?

okay so formally , till now it was your brain which was finding some words more catchy than others right ? this is what we wanna make our ml models do asw ? sounds cool ! (at least to me)

attention guys and girls ! , we are going to pay attention on attention of math , oops , math of attention :)

imagine i am in my lab , in fact i am , and i am gonna search for my lost octane pen ... , what can be my thought process ?

i will go to the senior guy sitting next to me and firstly ask him , "sir , have you seen my octane pen anywhere , by any chance ?" ...... , to some other senior , the same question , or "query" again , right ??

(ps : you must have been thinking that why this lad doesn't ask his juniors , answer is ,i don't have juniors :( , oh there is a chance of finding a second kira , ryuzaki confirms... , context : death note anime , a must watch )

so basically i am querying everyone and the "key" to "query" is octane , or a pen , right ? and what answers i got till now "no bro" , that's the "value" to my query , sad lol :(

key , query and value

imo , y'all must have got a context on what i am doing , i feel so , i may be wrong too !

so any sentence when fed to a dumb thingy (computer xd) is fed in terms of tokens , you can assume for the sake of simplicity that every word of a sentence $S$ is a token $t$ .

so sentence must be having many tokens or do you talk in acronyms lol ? at least not me , hence my sentence looks like this to be honest :

S = t_{1}, t_{2}, . . . . . . t_{n}

so this dumbo (on which am writing this blog) , reads these tokens in super dimensional space ,it's dumb after all , tbh we give it like this lol

t_{i} = [m_{1}, m_{2}, . . . . ., m_{k}]

hmmm , so for every token we have some pair of 3 vectors called as $Q$ , $K$ and $V$ , popularly known as query , key and value vector respectively.

what am I even trying to do ?

i am basically trying to calculate how much attention a word (token) needs to be given , basically a sort of rating i wanna give to a word , or trying to find how important it is given the whole sentence.

this computer also assigns $Q$ , $K$ and $V$ as well to some higher dimensional space like this :

Q = [q_{1}, q_{2}, . . . . q_{k}]

K = [k_{1}, k_{2}, . . . ., k_{k}}

V = [v_{1}, v_{2}, . . . . ., v_{m}]

do notice that $V$ lives in some other dimensions as opposed to the holy brother pair $Q$ and $K$ . i mean this wasn't my proposal , am narrating the holy story of attention is all you need , ik what you are thinking :p

what lies next ?

(we still didn't find the second kira , ryuzaki has employed lawliet on the mission)

what do you do with two vectors ? don't do a cross product , we need to churn out a score from these vectors , as score is a scalar ? is it really scalar , idk , think senpai !

in our context, it is . so basically on a serious note , we need to do a inner (dot) product between $Q$ and $K$ to find a score , it seems quite logical to do that asw , because you are trying to query on a key vector , so each component of $Q$ interacts (multiplies) with corresponding components (keys) and we add them up to get a cumulative information , sounds obvious , your answer may be no asw xD

so all we have done till now is doing dot product and it looks like this :

s c o r e = Q K^{T}

transpose is there just because we are taking dot product between matrices , i hope you get it , if you don't then my bro (makes a cute face) yayyy.

too big numbers ? let's take a life-line

what if the dot product gives us too large numbers , it won't feel cute any more that's scary , so I have just called my friend "softmax" , he will rescue us as we have gained so much weight lol

s c o r e = s o f t m a x (Q K^{T})

oh we forgot V !

$Q$ and $K$ have always been brothers , they lived in same super dimensions, even softmax helped them but our $V$ is again left out , lmao.

remember when we were searching for that octane pen? when someone said "yeah, I saw it on the blue desk", that "blue desk" is like our V vector - it's the actual information we care about! think about it this way:

$Q$ and $K$ helped us figure out who to pay attention to but $V$ tells us what information to actually remember

it's like when you're gossiping (which I totally don't do in the lab, wink wink):

"have you seen my pen?" (that's your $Q$ )
"oh yeah, I know about your pen" (that's how well $K$ matches)
"it's on the blue desk" (that's the valuable $V$ information!)

so let's churn it up and include $V$ with us,

s c o r e = s o f t m a x (Q K^{T}) V

wait , it exploded , nooo ,it vanished !

when this dumboo machine trains on the data , then it faces the exploding and vanishing gradients problem which we as a doctor want to cure it from , so we rather give it a medicine $k$ , this is how he looks like after scaling , jollyyyy !

s c o r e = s o f t m a x (\frac{Q K^{T}}{\sqrt{k}}) V

time to give it a name

score looks a bad name , what score bro , it isn't what we need to pay attention on ? oh , "attention" looks cool.

a t t e n t i o n = s o f t m a x (\frac{Q K^{T}}{\sqrt{k}}) V

that's it , we achieved it finally

self attention

on a serious note , self-attention calculates the attention of a particular token with respect to all the other tokens , and itself too , in the same sentence.

pheewwwwwww , it was a long blog to read , am sure , you all are bored , now that we understand self-attention from the ground up, we're ready to dive into the world of transformers. but that's a story for another sunny day in jodhpur, preferably with another cup of coffee and no lectures to attend.

until then, keep paying attention! (sorry, couldn't resist that one)

ps: still haven't found my octane pen. if anyone sees it, you know where to find me - in the lab, probably writing another blog post about machine learning while using a borrowed pen.

good bye !