Learning Transformer — ASO

By Sungjun Kim on AppTrack Ninja.

Attention, one step at a time

A Transformer, built by hand and shown one step at a time. Twenty-three screens follow "Attention Is All You Need" (Vaswani et al., 2017) in the paper's own order: a word becoming numbers, the three vectors, the scores, the softmax, four heads merged into one, the mask, the second attention, and the loop that writes one word at a time. What this is not: a trained model. Every weight here was set by hand, so that each one can be read in a single line. Nine numbers stand for a word instead of 512. One encoder layer and one decoder layer instead of six of each. Four heads, four words of vocabula…

Categories: Education, Reference

Keyword visibility, ranked search terms and daily position movement for Learning Transformer.