Machine Learning
33 posts · all posts
The Vision Transformer reads images as sequences of patches
How a Transformer reads a photo: every step from pixels to a sequence of 197 vectors on a real image, written from scratch and checked against Google's ViT-B/16. What the model learned about patches and positions, where its heads look, DINO's heads finding objects without labels, and ViT against a CNN on satellite images when labels are scarce.
AI crypto trading bots: what machine learning and LLMs can and can't do
AI trading bots and LLM agents tested on six years of Binance data: what machine learning predicts, what trading fees eat, and why LLM backtests lie
Beyond attention with Mamba, state space models and hybrids
State space models from the continuous equation up: discretisation, Mamba's selective scan written from scratch and checked against mamba-130m, what its time steps choose to remember, generation with a state of fixed size, Mamba-2 as masked attention, Mamba against Pythia on the same data, hybrids, and what selectivity buys, learned from scratch on selective copying.
Long context with RoPE scaling, YaRN and FlashAttention
Why a language model breaks past the length it was trained on, and how RoPE scaling fixes it. Position interpolation, NTK scaling, YaRN and Llama 3.1's scaling written from scratch and checked against transformers, raced on TinyLlama; FlashAttention built from the online softmax; Qwen3 finding a needle in a novel up to 64k tokens.
The KV cache budget and DeepSeek's latent attention
What one token costs in the KV cache of ten models, and how DeepSeek's multi-head latent attention cuts it. MLA written from scratch in both its forms, why rotary positions need their own dimensions, MiniCPM3-4B with its official weights, a latent cache against a rebuilt one on a T4, Qwen3 compressed without training, and a quantized cache.
Mixture-of-experts LLMs from scratch with OLMoE
How a dozen production mixtures of experts are built, from Mixtral to gpt-oss. The router written four ways, OLMoE in PyTorch with its official weights, the experts that English, Python, maths and Russian use, a model pruned for one kind of text, and an expert cache measured on real routing.
Modern LLM architecture and Qwen3 from scratch
What Mistral, Gemma and Qwen changed after Llama, read from a dozen configs. Grouped-query attention, sliding windows and attention sinks, QK-norm and soft-capping, Qwen3-0.6B written from scratch with its official weights, and LoRA that turns voice commands into JSON in English and Russian.
How Karl Friston's free energy principle works and what its critics say
Karl Friston's free energy principle with exact formulas and plain words. Surprisal and variational free energy, its link to the ELBO of machine learning, predictive coding on a worked example, active inference, Markov blankets, the experimental evidence and the main criticisms
How similar is the human brain to AI models?
A decade of neuroscience has compared brains with neural networks. Language models predict the brain's language network, vision models the visual cortex and reinforcement learning the dopamine signal, while memory, bodies, energy and lifelong learning are where the two still differ most
T5 and BART: the encoder–decoder, and a model that writes SQL
T5 written from scratch and loaded with Google's weights: relative position buckets, cross-attention, span corruption and BART's text infilling, and a T5 fine-tuned on Spider to write SQL, checked by running every query on SQLite.
GPT from scratch: GPT-2, the KV cache and a model that learns chess
GPT-2 written from scratch and loaded with OpenAI's weights, generation with a KV cache, the logit lens, and a small GPT trained on Lichess games that learns to play legal chess and keeps a board inside, found with a linear probe.
Embeddings and search: Sentence-BERT, cross-encoders and ColBERT
How a search engine reads: BM25 written from scratch, Sentence-BERT's bi-encoder trained with in-batch negatives, a cross-encoder reranker and ColBERT's late interaction, each checked against the real weights and put together into a literature search over 64,000 machine-learning papers.
From RoBERTa to ModernBERT: what seven years did to BERT
What seven years did to BERT: RoBERTa's training recipe, ELECTRA's replaced-token detection, DeBERTa's disentangled attention and ModernBERT's LLM-era block, each written in PyTorch and checked against the real weights, and the four compared on triaging GitHub issues of projects they never saw.
BERT from scratch: masked language modelling and fine-tuning
BERT written in PyTorch from scratch and loaded with Google's weights: why reading in both directions needed a mask, what BERT knows and what its attention heads do, and a detector of personal data in long financial documents, against regular expressions.
Mixture of experts: why the largest LLMs use a few percent of their weights per token
A mixture-of-experts layer in PyTorch trained next to dense models: the router, load balancing, dropped tokens, what experts learn, and serving
Hybrid attention: why new LLMs keep full attention in only a few layers
Linear attention, the delta rule and Gated DeltaNet in PyTorch, a recall test, GPU timings, and why Qwen3-Next and Kimi Linear keep few attention layers
It works on my laptop: a reproducible Docker image of a model
Ship a model so that it runs the same next year: a lock file with hashes, an image pinned by digest, builds that give the same bytes twice, GitHub Actions publishing to GHCR, the image pulled and run without Docker, and the model exported to ONNX for a free in-browser demo.
Serving a Transformer with FastAPI and Gradio: latency under load
Put a Transformer behind an HTTP API and measure it the way users meet it: a KV cache in beam search, int8 quantization that doesn't pay off, FastAPI and asyncio, open-loop load tests, queueing theory checked against the measurements, dynamic batching, and a Gradio page.
Your notebook is not a product: a Python package with tests and CI
Turn a model from a notebook into a Python package anyone can install: the src layout, pyproject.toml, a command line, four kinds of tests checked by mutation testing, GitHub Actions, and a model card that measures what the model can't do.
Is Centaur a model of the mind? An LLM fine-tuned on 160 psychology experiments
In 1973 Allen Newell asked psychology for one system that performs all its tasks. In 2025 a Llama fine-tuned on 160 experiments was presented as that system. What Centaur predicts, what happens when you take away its instructions, and why its own authors went on to beat it with half a million parameters
Hyperparameter tuning on a free GPU with W&B Sweeps
Spend an hour of a free GPU on hyperparameters: experiment tracking with Weights & Biases, a learning-rate range test, random search and successive halving from scratch, and W&B Sweeps with Bayesian optimization.
What Llama changed in the Transformer: RoPE, RMSNorm, SwiGLU
Pre-norm, RMSNorm, SwiGLU and RoPE: the changes Llama made to the 2017 Transformer, written in PyTorch, added to our translator one at a time and measured against what a new random seed does alone, with fused attention and a KV cache.
Evaluating machine translation with chrF and COMET
Evaluate English–Russian translation with BLEU, chrF and COMET written from scratch: multiple references, agreement with professional translators, significance tests and a contaminated test set.
Training a Transformer: warm-up, label smoothing, beam search, sampling
Train a Transformer on half a million sentence pairs on a free GPU, then decode it with greedy search, beam search and sampling.
Implementing the Transformer in PyTorch: why attention divides by √d_k
Attention, masks, heads, positions and the full encoder-decoder of the 2017 paper, written line by line in PyTorch and checked against real models.
How language models read: tokenization and BPE from scratch
What GPT-4o actually receives when you type, why Russian costs more tokens than English, and byte-pair encoding written from scratch.
LLM inference optimization: where the time and memory go
Why generating text is limited by memory rather than arithmetic, measured on a free GPU, and what serving engines like vLLM do about it: the KV cache, continuous batching, quantization and speculative decoding
Parameter-efficient fine-tuning in PyTorch: adapters, LoRA and the rest
How to adapt a pretrained transformer by training a fraction of a percent of it: what adapters, LoRA, prefix and prompt tuning, BitFit, (IA)³ and Compacter add to the model, written out in PyTorch and measured on one task
Building a Transformer in PyTorch, layer by layer
Positional encoding, self-attention, multi-head attention and cross-attention, then the encoder and decoder layers, written in PyTorch one module at a time.
Byte-pair encoding: writing a BPE tokenizer in Python
Byte-pair encoding written in Python step by step, trained on English–Russian sentences from Tatoeba and compared with the tokenizers of Hugging Face.
Can AI be creative?
What happens in a brain that has a new idea, where recombination stops, and what the measurements of machine creativity show, from AlphaGo's move 37 to GPT-4 on the Alternate Uses Task
The algebra of LoRA: how low-rank adaptation works
Why a fine-tuning update can be the product of two thin matrices: rank, the singular value decomposition and the Eckart–Young theorem, then LoRA's initialization, gradients and scale, derived by hand and checked on BERT
How large language models work, for those who missed out
What a large language model is, how it is trained in three stages, and what it could and could not do in spring 2024: a tour after Andrej Karpathy's talk, with the claims checked against their sources and measured on small open models