Transformer architectures from BERT to diffusion

Fifteen models that grew out of the Transformer, one per lesson. Each time we write the key part in PyTorch, load the real weights into it to check that it is the real thing, and put the model to work on a task it does well.

14 of 15 lessons out
Lessons
14 of 15 out
Reading
About 12 hours
Notebooks
Kaggle or Colab
Level
Intermediate
Price
Free
Code
MIT licence

About the course

The lessons go from encoders that read text to models that see, listen and draw. Each one starts from a limit of an earlier model and shows how the new one removed it.

Every lesson follows the same plan. The story of the model comes first, then its key part written from scratch, with the official weights loaded into that code and matched to the reference implementation. A small experiment on a free T4 shows the idea at work, and a practical task uses a different dataset each time.

At the end of each lesson the model gets the same card as the ones before it, with its objective, attention, positions, normalisation and sizes, so the family can be compared side by side.

Before you start

Start here after the first course, or if you can already write attention and a training loop in PyTorch yourself. Most lessons need a free T4 on Kaggle or Colab, and two of them use Kaggle's pair of T4s.

Lessons

Part 1 · Encoders that understand text · 3 lessons

  1. BERT from scratch with masked language modelling and fine-tuning

    BERT written in PyTorch and loaded with Google's weights. Why reading in both directions needed a mask, what BERT knows, and a detector of personal data in long financial documents, set against regular expressions.

    48 min read Kaggle Colab

  2. How BERT changed from RoBERTa to ModernBERT

    What seven years of research changed in BERT, from RoBERTa's training recipe to ModernBERT's block. Each model is checked against its real weights, and all four sort GitHub issues from projects they never saw.

    35 min read Kaggle Colab

  3. Semantic search with Sentence-BERT, cross-encoders and ColBERT

    BM25 from scratch, a Sentence-BERT bi-encoder trained on in-batch negatives, a cross-encoder reranker and ColBERT. Together they become a search engine over 58,000 machine-learning papers.

    48 min read Kaggle Colab

Part 2 · Decoders and the encoder–decoder · 3 lessons

  1. GPT-2 from scratch and a small GPT that learns chess

    GPT-2 written from scratch with OpenAI's weights, the KV cache and the logit lens. Then a small GPT trained on Lichess games learns legal chess, and a linear probe finds the board inside it.

    56 min read Kaggle Colab

  2. T5 and BART from scratch and a T5 that writes SQL

    T5 written from scratch with Google's weights, with its relative positions, cross-attention and span corruption, next to BART's text infilling. A T5 fine-tuned on Spider writes SQL, and every query is run on SQLite to check it.

    62 min read Kaggle Colab

  3. Modern LLM architecture and Qwen3 from scratch

    What Mistral, Gemma and Qwen changed after Llama, read from their configs. Qwen3-0.6B written from scratch with its official weights, and LoRA that turns voice commands into JSON in English and Russian.

    69 min read Kaggle Colab

Part 3 · Scale and efficiency · 4 lessons

  1. Mixture-of-experts LLMs from scratch with OLMoE

    How mixtures of experts from Mixtral to gpt-oss are built. OLMoE with its official weights shows which experts English, Python, maths and Russian use, and then gets pruned for one kind of text.

    39 min read Kaggle Colab

  2. The KV cache budget and DeepSeek's latent attention

    What one token costs in the KV cache of ten models, and how DeepSeek's latent attention cuts it. MLA in both of its forms, MiniCPM3-4B with its official weights, and Qwen3 compressed without training.

    36 min read Kaggle Colab

  3. Long context with RoPE scaling, YaRN and FlashAttention

    Why a language model breaks past the length it was trained on, and how RoPE scaling fixes it. YaRN and Llama 3.1's scaling compared on TinyLlama, FlashAttention built from the online softmax, and Qwen3 searching a novel for a needle.

    60 min read Kaggle Colab

  4. Mamba and state space models instead of attention

    State space models from the continuous equation to Mamba's selective scan, checked against mamba-130m. Mamba against Pythia on the same data, Mamba-2 as masked attention, and the layouts of hybrid models.

    52 min read Kaggle Colab

Part 4 · Beyond text · 5 lessons

  1. The Vision Transformer reads images as sequences of patches

    Every step from a photo to the 197 vectors the encoder reads, checked against Google's ViT-B/16. Where its heads look, how DINO finds objects without labels, and ViT against a CNN on satellite images with few labels.

    40 min read Kaggle Colab

  2. CLIP and SigLIP put images and text in one space

    Both towers of CLIP written from scratch with OpenAI's weights, and SigLIP's sigmoid loss next to the contrastive one. Zero-shot geolocation over 211 countries, the modality gap, and a word on a photo that changes the answer.

    64 min read Kaggle Colab

  3. How vision-language models read images, from LLaVA to Qwen2-VL

    LLaVA's tiles, projector and splice written from scratch and matched to LLaVA-OneVision. What an image costs in tokens, Qwen2-VL's dynamic resolution, and receipts and charts read under shrinking token budgets.

    58 min read Kaggle Colab

  4. Whisper turns speech into text with an encoder–decoder Transformer

    Whisper's log-mel front end, encoder and decoder written from scratch and checked against whisper-tiny. Word error rates in English and Russian, subtitle credits written over silence, and word timings from cross-attention.

    70 min read Kaggle Colab

  5. Diffusion Transformers from DiT to Stable Diffusion 3

    Denoising diffusion and flow matching, the adaLN-Zero block, a small DiT trained on butterflies, and the MM-DiT of Stable Diffusion 3.

    Coming

Reference and BibTeX

Koroleva, E. (2026). Transformer architectures from BERT to diffusion [Online course]. https://blog.korolevalab.com/teaching/transformer-architectures/

@misc{koroleva2026transformerarchitectures,
  author       = {Koroleva, Eva},
  title        = {{Transformer architectures from BERT to diffusion}},
  year         = {2026},
  howpublished = {Online course},
  url          = {https://blog.korolevalab.com/teaching/transformer-architectures/}
}