The Library / Intelligence & systems

Attention Is All You Need

Ashish Vaswani et al.

The paper that introduced the Transformer architecture.

Read it ↗

Worth keeping

  1. 1

    The Transformer processes sequences using attention without relying on recurrence or convolution.

  2. 2

    Attention lets the model weight relationships among different positions in a sequence.

  3. 3

    The architecture enabled more parallel training and demonstrated strong results on machine translation.

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

In Not Boring4 essays that cite it