Transformers from scratch to production

We write an English–Russian translator in PyTorch, starting from the tokenizer, and carry it all the way to a Docker image that anyone can run.

9 of 10 lessons out
Lessons
9 of 10 out
Reading
About 8 hours
Notebooks
Kaggle or Colab
Level
Beginner
Price
Free
Code
MIT licence

About the course

The course follows one model from its first line of code to a running service. You write the tokenizer and the Transformer yourself, train them on half a million sentence pairs and check the translations with metrics that are themselves checked against professional translators.

Then the model gets what came after 2017: the changes Llama made, a tuned learning rate, a Python package with tests, an HTTP API measured under load and a Docker image that rebuilds to the same bytes.

Each lesson starts by downloading what the earlier ones produced, the tokenizer and the checkpoints, so any of them can be opened on its own.

Before you start

You need to read Python; the first lesson also uses a little pandas. Some PyTorch helps from lesson 2 on, but every step is explained. You don't need a GPU of your own: the lessons that train run on a free T4 on Kaggle or Colab, the rest on a CPU.

Lessons

Part 1 · Build the model · 3 lessons

  1. Tokenization and BPE from scratch

    What GPT-4o and Llama 3 receive when you type, and why Russian costs more tokens than English. We clean the parallel data, split it without leaks and write byte-pair encoding by hand.

    51 min read Kaggle Colab

  2. A Transformer from scratch in PyTorch

    The encoder–decoder of the 2017 paper, from attention and masks to positions, with every part checked against real models. Along the way you'll see why attention divides by √d_k.

    51 min read Kaggle Colab

  3. Training a Transformer and decoding with beam search

    The translator learns from half a million sentence pairs on a free GPU. We look at what warm-up and label smoothing do, then compare greedy search, beam search and sampling on its output.

    60 min read Kaggle Colab

Part 2 · Measure and improve it · 3 lessons

  1. Evaluating machine translation with chrF and COMET

    BLEU and chrF written from scratch and COMET run by hand, then compared with the judgments of professional translators. The paired bootstrap tells a real gain from noise, and a test set seen in training shows what leakage does to the scores.

    66 min read Kaggle Colab

  2. What Llama changed in the Transformer with RoPE, RMSNorm and SwiGLU

    Pre-norm, RMSNorm, SwiGLU and RoPE go into the translator one at a time. Each change is measured against what a new random seed does on its own, and fused attention and a KV cache come last.

    48 min read Kaggle Colab

  3. Hyperparameter tuning on a free GPU with W&B Sweeps

    One hour of a free GPU spent on hyperparameters. We write a learning-rate range test, random search and successive halving, then hand the search to W&B Sweeps and check the gain against seeds.

    25 min read Kaggle Colab

Part 3 · Ship it · 3 lessons

  1. Turning a notebook into a Python package with tests and CI

    The model leaves the notebook and becomes a package anyone can install, with a command line, tests checked by mutation testing and CI on GitHub Actions. Its model card lists what it can't do.

    72 min read Kaggle Colab

  2. Serving a Transformer with FastAPI and measuring it under load

    The translator goes behind an HTTP API and is measured the way its users meet it. Open-loop load tests, queueing theory checked against the numbers, dynamic batching, and int8 quantization that doesn't pay off.

    61 min read Kaggle Colab

  3. A reproducible Docker image for a machine learning model

    An image that builds to the same bytes twice and runs the same next year. Lock files with hashes, digests, CI that rebuilds and compares, and an ONNX export for a demo in the browser.

    51 min read Kaggle Colab

Part 4 · A better translator · 1 lesson

  1. Making the translator better with distillation and more data

    Which helps a small translator most, distillation from a bigger model, synthetic data from its domain, a bigger student or an ensemble. NLLB and LLMs serve as reference points.

    Coming

Reference and BibTeX

Koroleva, E. (2025). Transformers from scratch to production [Online course]. https://blog.korolevalab.com/teaching/transformers-from-scratch/

@misc{koroleva2025transformersfromscratch,
  author       = {Koroleva, Eva},
  title        = {{Transformers from scratch to production}},
  year         = {2025},
  howpublished = {Online course},
  url          = {https://blog.korolevalab.com/teaching/transformers-from-scratch/}
}