Tutorials & guidesImpact: Medium
Oleksandr Kuzmenko, EditorPublished 5 min readAI-assisted · editor-reviewed
TL;DR
- Transformers require position signals because attention matrix operations are inherently permutation-invariant.
- GPT-2 relies on learned absolute position tables added to token embeddings before the first transformer block.
- Modern models such as MiniMax-M3 inject position information within attention layers via RoPE.
- Causal masks provide implicit ordering but cannot distinguish the magnitude of separation between tokens, making explicit positional methods essential.
Key facts
- Modern Primary Encoding
- Rotary Position Embedding (RoPE)
- Legacy Benchmark Architecture
- GPT-2 learned absolute position lookup tables
- MiniMax-M3 Architecture Detail
- 60 transformer layers (first 3 dense, 57 MoE)
- Typical Vocabulary Size Range
- 50,000 to 200,000 token entries
Absolute vs. Relative Encodings. Transformers require position signals because matrix multiplications in attention layers are naturally order-blind. Without position vectors, sentences with identical words in different orders yield indistinguishable representations. Learned Absolute Position Embeddings (GPT-2): A fixed lookup table is learned during training and summed element-wise into input token embeddings before entering the first transformer block. Rotary Position Embedding (RoPE): Applied inside every attention block directly to query and key projections.
Architectural Divergence. Modern open LLMs balance these mechanisms across network depths. In architectures like MiniMax-M3, the first 3 of its 60 feedforward layers are dense while the remaining 57 employ Mixture of Experts, yet RoPE is applied inside attention. Regarding NoPE (No Positional Embeddings), while causal attention supplies implicit ordering by restricting attention to prior tokens, it does not distinguish a separation of one position from a separation of ten, making explicit positional methods still useful or essential.
#PyTorch#GPT-2#MiniMax-M3
