Transformer architectures from BERT to diffusion
Fifteen models that grew out of the Transformer, one per lesson. Each time we write the key part in PyTorch, load the real weights into it to check that it is the real thing, and put the model to work on a task it does well.
- Lessons
- 14 of 15 out
- Reading
- About 12 hours
- Notebooks
- Kaggle or Colab
- Level
- Intermediate
- Price
- Free
- Code
- MIT licence
About the course
The lessons go from encoders that read text to models that see, listen and draw. Each one starts from a limit of an earlier model and shows how the new one removed it.
Every lesson follows the same plan. The story of the model comes first, then its key part written from scratch, with the official weights loaded into that code and matched to the reference implementation. A small experiment on a free T4 shows the idea at work, and a practical task uses a different dataset each time.
At the end of each lesson the model gets the same card as the ones before it, with its objective, attention, positions, normalisation and sizes, so the family can be compared side by side.
Before you start
Start here after the first course, or if you can already write attention and a training loop in PyTorch yourself. Most lessons need a free T4 on Kaggle or Colab, and two of them use Kaggle's pair of T4s.
Lessons
Part 1 · Encoders that understand text · 3 lessons
-
BERT from scratch with masked language modelling and fine-tuning
BERT written in PyTorch and loaded with Google's weights. Why reading in both directions needed a mask, what BERT knows, and a detector of personal data in long financial documents, set against regular expressions.
-
How BERT changed from RoBERTa to ModernBERT
What seven years of research changed in BERT, from RoBERTa's training recipe to ModernBERT's block. Each model is checked against its real weights, and all four sort GitHub issues from projects they never saw.
-
Semantic search with Sentence-BERT, cross-encoders and ColBERT
BM25 from scratch, a Sentence-BERT bi-encoder trained on in-batch negatives, a cross-encoder reranker and ColBERT. Together they become a search engine over 58,000 machine-learning papers.
Part 2 · Decoders and the encoder–decoder · 3 lessons
-
GPT-2 from scratch and a small GPT that learns chess
GPT-2 written from scratch with OpenAI's weights, the KV cache and the logit lens. Then a small GPT trained on Lichess games learns legal chess, and a linear probe finds the board inside it.
-
T5 and BART from scratch and a T5 that writes SQL
T5 written from scratch with Google's weights, with its relative positions, cross-attention and span corruption, next to BART's text infilling. A T5 fine-tuned on Spider writes SQL, and every query is run on SQLite to check it.
-
Modern LLM architecture and Qwen3 from scratch
What Mistral, Gemma and Qwen changed after Llama, read from their configs. Qwen3-0.6B written from scratch with its official weights, and LoRA that turns voice commands into JSON in English and Russian.
Part 3 · Scale and efficiency · 4 lessons
-
Mixture-of-experts LLMs from scratch with OLMoE
How mixtures of experts from Mixtral to gpt-oss are built. OLMoE with its official weights shows which experts English, Python, maths and Russian use, and then gets pruned for one kind of text.
-
The KV cache budget and DeepSeek's latent attention
What one token costs in the KV cache of ten models, and how DeepSeek's latent attention cuts it. MLA in both of its forms, MiniCPM3-4B with its official weights, and Qwen3 compressed without training.
-
Long context with RoPE scaling, YaRN and FlashAttention
Why a language model breaks past the length it was trained on, and how RoPE scaling fixes it. YaRN and Llama 3.1's scaling compared on TinyLlama, FlashAttention built from the online softmax, and Qwen3 searching a novel for a needle.
-
Mamba and state space models instead of attention
State space models from the continuous equation to Mamba's selective scan, checked against mamba-130m. Mamba against Pythia on the same data, Mamba-2 as masked attention, and the layouts of hybrid models.
Part 4 · Beyond text · 5 lessons
-
The Vision Transformer reads images as sequences of patches
Every step from a photo to the 197 vectors the encoder reads, checked against Google's ViT-B/16. Where its heads look, how DINO finds objects without labels, and ViT against a CNN on satellite images with few labels.
-
CLIP and SigLIP put images and text in one space
Both towers of CLIP written from scratch with OpenAI's weights, and SigLIP's sigmoid loss next to the contrastive one. Zero-shot geolocation over 211 countries, the modality gap, and a word on a photo that changes the answer.
-
How vision-language models read images, from LLaVA to Qwen2-VL
LLaVA's tiles, projector and splice written from scratch and matched to LLaVA-OneVision. What an image costs in tokens, Qwen2-VL's dynamic resolution, and receipts and charts read under shrinking token budgets.
-
Whisper turns speech into text with an encoder–decoder Transformer
Whisper's log-mel front end, encoder and decoder written from scratch and checked against whisper-tiny. Word error rates in English and Russian, subtitle credits written over silence, and word timings from cross-attention.
-
Diffusion Transformers from DiT to Stable Diffusion 3
Denoising diffusion and flow matching, the adaLN-Zero block, a small DiT trained on butterflies, and the MM-DiT of Stable Diffusion 3.
Reference and BibTeX
Koroleva, E. (2026). Transformer architectures from BERT to diffusion [Online course]. https://blog.korolevalab.com/teaching/transformer-architectures/
@misc{koroleva2026transformerarchitectures,
author = {Koroleva, Eva},
title = {{Transformer architectures from BERT to diffusion}},
year = {2026},
howpublished = {Online course},
url = {https://blog.korolevalab.com/teaching/transformer-architectures/}
}