![\"https://i.ibb.co/18yGFcC/Selection-999-044.png\"](\"https://i.ibb.co/18yGFcC/Selection-999-044.png\")
answer1: \" would you want to be my transformer?\"
\nanswer2: \"attention is all you need\"
answer1: \" would you want to be my transformer?\"
\nanswer2: \"attention is all you need\"
Nice. I found that many tutorials jump directly to Q,K,V without explanation what attention is and how it works. I really like explanations from this video: https://youtu.be/yGTUuEx3GkA
", "rawMarkdown": "Nice. I found that many tutorials jump directly to Q,K,V without explanation what attention is and how it works. I really like explanations from this video: https://youtu.be/yGTUuEx3GkA ", "votes": 1 }, { "id": 2129348, "postDate": "2023-02-04T14:42:54.620Z", "content": "![\"Selection-999-045\"](\"https://i.ibb.co/FBzt4DS/Selection-999-045.png\")
\n![\"Selection-999-046\"](\"https://i.ibb.co/809Wvw0/Selection-999-046.png\")
![\"Selection-999-047\"](\"https://i.ibb.co/XYjkxmy/Selection-999-047.png\")
![\"Selection-999-050\"](\"https://i.ibb.co/vjk6pNk/Selection-999-050.png\")
![\"Selection-999-048\"](\"https://i.ibb.co/34TW26B/Selection-999-048.png\")
\nsome insights in explaining transformer.
\n\"telling data science story is a great way to teach data science concept\"
\n\n\n\n\n---\n\n\n \n\n\nsome insights in explaining transformer.\n\n\"telling data science story is a great way to teach data science concept\"",
"votes": 2,
"replies": [
{
"id": 2129356,
"postDate": "2023-02-04T14:57:03.333Z",
"content": "![\"Selection-999-051\"](\"https://i.ibb.co/921Mm3w/Selection-999-051.png\")
![\"Selection-999-054\"](\"https://i.ibb.co/hHmMt0p/Selection-999-054.png\")
i get rejected by gpt3 !!!
\n![\"Selection-999-061\"](\"https://i.ibb.co/HCrqkDZ/Selection-999-061.png\")
![\"Selection-999-060\"](\"https://i.ibb.co/6WqK7F1/Selection-999-060.png\")
\n\n![\"Selection-999-053\"](\"https://i.ibb.co/p16wVSd/Selection-999-053.png\")
\n\n\n![\"Selection-999-055\"](\"https://i.ibb.co/ZLTTRg9/Selection-999-055.png\")
\n\nThanks for sharing.
\nCan I assume that the Q(uery) and the K(ey) are equivalent in vision transformer? For example, the Q and the K of the early transformer stage are poor but in the late stage they become superior, like the process of a person's growth.
", "rawMarkdown": "Thanks for sharing. \n\nCan I assume that the Q(uery) and the K(ey) are equivalent in vision transformer? For example, the Q and the K of the early transformer stage are poor but in the late stage they become superior, like the process of a person's growth.", "replies": [ { "id": 2131485, "postDate": "2023-02-06T06:45:06.170Z", "content": "should be similar to CNN Hierarchical representation
\nhttps://miro.medium.com/max/720/1*bIY_8FE7dJaGk1kjU7iVSg.webp
you can visualize Q,K from high attention patches at different layer.
\nhigh attention means a matched, i.e. the things being looked for
got it. Thanks!
", "rawMarkdown": "got it. Thanks!" }, { "id": 2131871, "postDate": "2023-02-06T13:01:05.380Z", "content": "![\"Selection-999-167\"](\"https://i.ibb.co/YZk4RrG/Selection-999-167.png\")
\n![\"Selection-999-166\"](\"https://i.ibb.co/Tts7GPr/Selection-999-166.png\")
i always thing of transformer as a giant decision tree of dynamic switches. At start the first query looked for certain fixed things. Then after attended by the keys (and modified by values), it dynamically changes what it wants to looked for.
", "rawMarkdown": "\n\n\ni always thing of transformer as a giant decision tree of dynamic switches. At start the first query looked for certain fixed things. Then after attended by the keys (and modified by values), it dynamically changes what it wants to looked for.\n\n ",
"votes": 1
},
{
"id": 2132570,
"postDate": "2023-02-06T21:54:01.527Z",
"content": "code for visually trasnformer. this reproduce the figures in the paper:
\nhttps://keras.io/examples/vision/probing_vits/
The things you recommended are always inspiring. Thanks a lot. I will try it later.
", "rawMarkdown": "The things you recommended are always inspiring. Thanks a lot. I will try it later." }, { "id": 2132811, "postDate": "2023-02-07T03:55:25.430Z", "content": "I add some own understandings in your figure. Green blocks represent what are looked for, and the orange represent what a certain image has.
\n![\"modified_fig\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2325440%2Fc9ae93922e401f4b4d3819f6455e30d1%2Fmodifiedfig.png?generation=1675742083914102&alt=media\")
kind of correct …. but :
\npredic = f_cls(y)
\ny = attent * v
\nattent = softmax(similarity(q,k))
k,v is from x
\nq is from cls
if you are interested, you can email me and i can do a presentation for you. these illustrations are actually from a mini-course i give, see attachment
", "rawMarkdown": "kind of correct .... but :\npredic = f\\_cls(y)\ny = attent * v\nattent = softmax(similarity(q,k))\n\nk,v is from x\nq is from cls\n\n\n\n\nif you are interested, you can email me and i can do a presentation for you. these illustrations are actually from a mini-course i give, see attachment" }, { "id": 2132940, "postDate": "2023-02-07T06:21:26.170Z", "content": "code of the presentation slide
", "rawMarkdown": "code of the presentation slide" } ] } ] }, { "id": 2129989, "postDate": "2023-02-05T04:35:17.543Z", "content": "@hengck23 Attention do be all you need. Man, this paper literally meme it ways to greatness haha
", "rawMarkdown": "@hengck23 Attention do be all you need. Man, this paper literally meme it ways to greatness haha" }, { "id": 2129605, "postDate": "2023-02-04T18:37:38.483Z", "content": "Wow, that's amazing to see. Kinda reminds me of Romeo and Juliet, but in the metaverse that we're in.
", "rawMarkdown": "Wow, that's amazing to see. Kinda reminds me of Romeo and Juliet, but in the metaverse that we're in." }, { "id": 2129375, "postDate": "2023-02-04T15:19:55.027Z", "content": "Hi hengck, I just started the competition, do you think I can do something before the deadline?
", "rawMarkdown": "Hi hengck, I just started the competition, do you think I can do something before the deadline?" } ], "comments": [ { "id": 2130128, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-02-05T06:59:39.810000", "content": "Nice. I found that many tutorials jump directly to Q,K,V without explanation what attention is and how it works. I really like explanations from this video: https://youtu.be/yGTUuEx3GkA
", "votes": 1, "replies": [] }, { "id": 2129348, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-04T14:42:54.620000", "content": "
\n
\nsome insights in explaining transformer.
\n\"telling data science story is a great way to teach data science concept\"
i get rejected by gpt3 !!!
\n
Thanks for sharing.
\nCan I assume that the Q(uery) and the K(ey) are equivalent in vision transformer? For example, the Q and the K of the early transformer stage are poor but in the late stage they become superior, like the process of a person's growth.
", "votes": 0, "replies": [ { "id": 2131485, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-06T06:45:06.170000", "content": "should be similar to CNN Hierarchical representation
\nhttps://miro.medium.com/max/720/1*bIY_8FE7dJaGk1kjU7iVSg.webp
you can visualize Q,K from high attention patches at different layer.
\nhigh attention means a matched, i.e. the things being looked for
got it. Thanks!
", "votes": 0, "replies": [] }, { "id": 2131871, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-06T13:01:05.380000", "content": "
\n
i always thing of transformer as a giant decision tree of dynamic switches. At start the first query looked for certain fixed things. Then after attended by the keys (and modified by values), it dynamically changes what it wants to looked for.
", "votes": 1, "replies": [] }, { "id": 2132570, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-06T21:54:01.527000", "content": "code for visually trasnformer. this reproduce the figures in the paper:
\nhttps://keras.io/examples/vision/probing_vits/
The things you recommended are always inspiring. Thanks a lot. I will try it later.
", "votes": 0, "replies": [] }, { "id": 2132811, "author_name": "gray98", "author_url": "", "post_date": "2023-02-07T03:55:25.430000", "content": "I add some own understandings in your figure. Green blocks represent what are looked for, and the orange represent what a certain image has.
\n
kind of correct …. but :
\npredic = f_cls(y)
\ny = attent * v
\nattent = softmax(similarity(q,k))
k,v is from x
\nq is from cls
if you are interested, you can email me and i can do a presentation for you. these illustrations are actually from a mini-course i give, see attachment
", "votes": 0, "replies": [] }, { "id": 2132940, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-07T06:21:26.170000", "content": "code of the presentation slide
", "votes": 0, "replies": [] } ] } ] }, { "id": 2129989, "author_name": "Minh T. Nguyen", "author_url": "", "post_date": "2023-02-05T04:35:17.543000", "content": "@hengck23 Attention do be all you need. Man, this paper literally meme it ways to greatness haha
", "votes": 0, "replies": [] }, { "id": 2129605, "author_name": "Dino Wun", "author_url": "", "post_date": "2023-02-04T18:37:38.483000", "content": "Wow, that's amazing to see. Kinda reminds me of Romeo and Juliet, but in the metaverse that we're in.
", "votes": 0, "replies": [] }, { "id": 2129375, "author_name": "Qing Cha", "author_url": "", "post_date": "2023-02-04T15:19:55.027000", "content": "Hi hengck, I just started the competition, do you think I can do something before the deadline?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2129187": "answer1: \" would you want to be my transformer?\"\nanswer2: \"attention is all you need\"\n\n\n\n", "2130128": "Nice. I found that many tutorials jump directly to Q,K,V without explanation what attention is and how it works. I really like explanations from this video: https://youtu.be/yGTUuEx3GkA ", "2129348": "\n\n\n\n\n\n---\n\n\n \n\n\nsome insights in explaining transformer.\n\n\"telling data science story is a great way to teach data science concept\"",
"2131302": "Thanks for sharing. \n\nCan I assume that the Q(uery) and the K(ey) are equivalent in vision transformer? For example, the Q and the K of the early transformer stage are poor but in the late stage they become superior, like the process of a person's growth.",
"2129989": "@hengck23 Attention do be all you need. Man, this paper literally meme it ways to greatness haha",
"2129605": "Wow, that's amazing to see. Kinda reminds me of Romeo and Juliet, but in the metaverse that we're in.",
"2129375": "Hi hengck, I just started the competition, do you think I can do something before the deadline?"
}
}