Transformers from scratch to production
We write an English–Russian translator in PyTorch, starting from the tokenizer, and carry it all the way to a Docker image that anyone can run.
- Lessons
- 9 of 10 out
- Reading
- About 8 hours
- Notebooks
- Kaggle or Colab
- Level
- Beginner
- Price
- Free
- Code
- MIT licence
About the course
The course follows one model from its first line of code to a running service. You write the tokenizer and the Transformer yourself, train them on half a million sentence pairs and check the translations with metrics that are themselves checked against professional translators.
Then the model gets what came after 2017: the changes Llama made, a tuned learning rate, a Python package with tests, an HTTP API measured under load and a Docker image that rebuilds to the same bytes.
Each lesson starts by downloading what the earlier ones produced, the tokenizer and the checkpoints, so any of them can be opened on its own.
Before you start
You need to read Python; the first lesson also uses a little pandas. Some PyTorch helps from lesson 2 on, but every step is explained. You don't need a GPU of your own: the lessons that train run on a free T4 on Kaggle or Colab, the rest on a CPU.
Lessons
Part 1 · Build the model · 3 lessons
-
Tokenization and BPE from scratch
What GPT-4o and Llama 3 receive when you type, and why Russian costs more tokens than English. We clean the parallel data, split it without leaks and write byte-pair encoding by hand.
-
A Transformer from scratch in PyTorch
The encoder–decoder of the 2017 paper, from attention and masks to positions, with every part checked against real models. Along the way you'll see why attention divides by √d_k.
-
Training a Transformer and decoding with beam search
The translator learns from half a million sentence pairs on a free GPU. We look at what warm-up and label smoothing do, then compare greedy search, beam search and sampling on its output.
Part 2 · Measure and improve it · 3 lessons
-
Evaluating machine translation with chrF and COMET
BLEU and chrF written from scratch and COMET run by hand, then compared with the judgments of professional translators. The paired bootstrap tells a real gain from noise, and a test set seen in training shows what leakage does to the scores.
-
What Llama changed in the Transformer with RoPE, RMSNorm and SwiGLU
Pre-norm, RMSNorm, SwiGLU and RoPE go into the translator one at a time. Each change is measured against what a new random seed does on its own, and fused attention and a KV cache come last.
-
Hyperparameter tuning on a free GPU with W&B Sweeps
One hour of a free GPU spent on hyperparameters. We write a learning-rate range test, random search and successive halving, then hand the search to W&B Sweeps and check the gain against seeds.
Part 3 · Ship it · 3 lessons
-
Turning a notebook into a Python package with tests and CI
The model leaves the notebook and becomes a package anyone can install, with a command line, tests checked by mutation testing and CI on GitHub Actions. Its model card lists what it can't do.
-
Serving a Transformer with FastAPI and measuring it under load
The translator goes behind an HTTP API and is measured the way its users meet it. Open-loop load tests, queueing theory checked against the numbers, dynamic batching, and int8 quantization that doesn't pay off.
-
A reproducible Docker image for a machine learning model
An image that builds to the same bytes twice and runs the same next year. Lock files with hashes, digests, CI that rebuilds and compares, and an ONNX export for a demo in the browser.
Part 4 · A better translator · 1 lesson
-
Making the translator better with distillation and more data
Which helps a small translator most, distillation from a bigger model, synthetic data from its domain, a bigger student or an ensemble. NLLB and LLMs serve as reference points.
Reference and BibTeX
Koroleva, E. (2025). Transformers from scratch to production [Online course]. https://blog.korolevalab.com/teaching/transformers-from-scratch/
@misc{koroleva2025transformersfromscratch,
author = {Koroleva, Eva},
title = {{Transformers from scratch to production}},
year = {2025},
howpublished = {Online course},
url = {https://blog.korolevalab.com/teaching/transformers-from-scratch/}
}