teaching
Transformers: from scratch to production
Build an English→Russian translator with a Transformer you write yourself, measure it honestly, improve it, and ship it as a tested package and a reproducible service.
Free GPU on Kaggle or Colab · every lesson runs on its own · open source
Lessons
- 01
How language models read: tokenization and BPE
Inside the tokenizers of GPT-4o and Llama 3, cleaning parallel data, leak-free splits, BPE by hand, Hugging Face tokenizers
- 02
Implementing the Transformer in PyTorch: why attention divides by √d_k
Real models' blueprints, attention and why it divides by √d_k, masks, heads, positions, the encoder–decoder, checks against PyTorch
- 03
Training a Transformer: warm-up, label smoothing, beam search, sampling
Token-based batches, label smoothing, warm-up and what happens without it, mixed precision, greedy, beam search and sampling
- 04
Evaluating machine translation with chrF and COMET
BLEU and chrF from scratch, COMET by hand, sacreBLEU signatures, metrics against human judgments, the paired bootstrap, test-set contamination
- 05
What Llama changed in the Transformer: RoPE, RMSNorm, SwiGLU
Llama 3.1 from its config, pre-norm, RMSNorm, SwiGLU, RoPE, fused attention, a KV cache, an ablation ladder against the noise of random seeds
- 06
Hyperparameter tuning on a free GPU with W&B Sweeps
Experiment tracking with W&B, a learning-rate range test, random search and successive halving from scratch, W&B Sweeps with Bayesian optimization
- 07
Your notebook is not a product: a Python package with tests and CI
The src layout, pyproject.toml, configs that refuse bad values, a command line, four kinds of tests, mutation testing, GitHub Actions, a model card
- 08
Serving a Transformer with FastAPI and Gradio: latency under load
Where inference time goes, the KV cache in beam search, int8 quantization, FastAPI and asyncio, load tests without coordinated omission, queueing theory checked by simulation, dynamic batching, Gradio
- 09
It works on my laptop: a reproducible Docker image of a model
Lock files with hashes, an image taken apart by hand (registries, manifests, layers, digests), reproducible builds, CI that rebuilds and compares, GitHub Container Registry, the image run without Docker, ONNX in the browser
- 10
Knowledge distillation, data and scale: what makes a translator better
Sequence-level distillation from opus-mt, synthetic in-domain data, a bigger student, checkpoint averaging and ensembles, NLLB and LLMs as reference points
soon
Transformer architectures: from BERT to diffusion
Fifteen models built on the Transformer, one lesson each: what it changed and why, its key parts written from scratch and checked against the real weights, its history, and a practical task it does well.
Free GPU on Kaggle or Colab · every lesson runs on its own · open source
Lessons
- 01
BERT from scratch: masked language modelling and fine-tuning
Why bidirectional needed a mask, WordPiece, the 80/10/10 mask, BERT-base's weights in our own encoder, what it knows and what its heads look at, a personal-data detector for long documents
- 02
From RoBERTa to ModernBERT: what seven years did to BERT
RoBERTa's recipe, DeBERTa's disentangled attention, ELECTRA's replaced-token detection, ModernBERT's local and global attention, three generations on one task
- 03
Embeddings and search: Sentence-BERT, cross-encoders and ColBERT
BM25 from scratch, why an encoder's own vectors can't search, in-batch negatives and InfoNCE, a model adapted to its corpus in ten minutes, a cross-encoder reranker, ColBERT's MaxSim, a hybrid search over 58,000 ML papers
- 04
GPT from scratch: GPT-2, the KV cache and a model that learns chess
GPT-1 to GPT-3, the GPT-2 block and OpenAI's weights in our code, the KV cache, the logit lens, a chess GPT trained on Lichess games, a 72-hour model transplanted, and the board found inside with a linear probe
- 05
T5 and BART: the encoder–decoder, and a model that writes SQL
Text-to-text, T5 in our code with Google's weights, relative position buckets, cross-attention that aligns German with English, span corruption and BART's infilling, a T5 that writes SQL for 1,034 Spider questions, every query checked on SQLite, and why decoder-only models won
- 06
Modern LLM architecture and Qwen3 from scratch
Fourteen model configs read side by side, grouped-query attention and the KV cache, small Transformers trained to diverge and saved by QK-norm, Qwen3 in PyTorch with the official weights, sliding windows and attention sinks, and LoRA that turns voice commands into JSON in English and Russian
- 07
Mixture-of-experts LLMs from scratch with OLMoE
Nine production mixtures read from their configs, the router written four ways, OLMoE in PyTorch with its official weights on two GPUs, which experts English, Python, maths and Russian use, a model pruned for one kind of text, and an expert cache measured on real routing
- 08
The KV cache budget and DeepSeek's latent attention
What a token costs in the cache of ten models, multi-head latent attention written in both forms, why rotary positions need their own dimensions, MiniCPM3-4B with its official weights, a latent cache against a rebuilt one on a T4, Qwen3 compressed without training, and a quantized cache
- 09
Long context with RoPE scaling, YaRN and FlashAttention
Eleven models' context windows read from their configs, TinyLlama's cliff at token 2,049 and the rotary pairs behind it, position interpolation, NTK, YaRN and Llama 3.1's scaling written from scratch and raced on books, FlashAttention built from the online softmax, and Qwen3-4B looking for a needle and answering questions about a whole novel
- 10
Beyond attention with Mamba, state space models and hybrids
A state space model from its differential equation, Mamba's selective scan written from scratch with mamba-130m's weights, what its steps choose to remember, the parallel scan, generation with a state of fixed size, Mamba-2 as masked attention, Mamba against Pythia, the hybrid layouts of seven models, and selective copying trained from scratch
- 11
The Vision Transformer reads images as sequences of patches
Every step from a photo to the 197 vectors the encoder reads, ViT-B/16 written from scratch with Google's weights, shuffled patches and learned positions, where the heads look, attention rollout, DINO's heads finding objects without labels, and ViT against a CNN and DINOv2 on satellite images with few labels
- 12
CLIP and SigLIP: one space for images and text
Contrastive and sigmoid losses, zero-shot classification, image search, typographic attacks
soon - 13
Vision-language models: from LLaVA to Qwen-VL
The projector, visual tokens and what they cost, dynamic resolution, a small model reading charts and receipts
soon - 14
Whisper: speech recognition as an encoder–decoder
The log-mel spectrogram from scratch, task tokens, timestamps, whisper-tiny in our code, word error rate in English and Russian, hallucinations on silence
soon - 15
Diffusion Transformers: DiT and Stable Diffusion 3
Denoising diffusion and flow matching, adaLN-Zero, a small DiT trained on butterflies, MM-DiT
soon