How a Transformer reads a photo: every step from pixels to a sequence of 197 vectors on a real image, written from scratch and checked against Google's ViT-B/16. What the model learned about patches and positions, where its heads look, DINO's heads finding objects without labels, and ViT against a CNN on satellite images when labels are scarce.
Polymarket's prices checked against what happened across almost three million resolved markets: how well calibrated they are, what accuracy figures hide, where longshots and favourites are mispriced, insiders, oracles and the cost of trading
AI trading bots and LLM agents tested on six years of Binance data: what machine learning predicts, what trading fees eat, and why LLM backtests lie
State space models from the continuous equation up: discretisation, Mamba's selective scan written from scratch and checked against mamba-130m, what its time steps choose to remember, generation with a state of fixed size, Mamba-2 as masked attention, Mamba against Pythia on the same data, hybrids, and what selectivity buys, learned from scratch on selective copying.
Why a language model breaks past the length it was trained on, and how RoPE scaling fixes it. Position interpolation, NTK scaling, YaRN and Llama 3.1's scaling written from scratch and checked against transformers, raced on TinyLlama; FlashAttention built from the online softmax; Qwen3 finding a needle in a novel up to 64k tokens.
What one token costs in the KV cache of ten models, and how DeepSeek's multi-head latent attention cuts it. MLA written from scratch in both its forms, why rotary positions need their own dimensions, MiniCPM3-4B with its official weights, a latent cache against a rebuilt one on a T4, Qwen3 compressed without training, and a quantized cache.
How a dozen production mixtures of experts are built, from Mixtral to gpt-oss. The router written four ways, OLMoE in PyTorch with its official weights, the experts that English, Python, maths and Russian use, a model pruned for one kind of text, and an expert cache measured on real routing.
What Mistral, Gemma and Qwen changed after Llama, read from a dozen configs. Grouped-query attention, sliding windows and attention sinks, QK-norm and soft-capping, Qwen3-0.6B written from scratch with its official weights, and LoRA that turns voice commands into JSON in English and Russian.
The hard problem of consciousness for readers used to physicalism. Thought experiments from Mary's room to fading qualia, positions from illusionism to panpsychism, the scientific theories, the first experiment that tested two of them against each other, and what they imply for AI
Karl Friston's free energy principle with exact formulas and plain words. Surprisal and variational free energy, its link to the ELBO of machine learning, predictive coding on a worked example, active inference, Markov blankets, the experimental evidence and the main criticisms
A decade of neuroscience has compared brains with neural networks. Language models predict the brain's language network, vision models the visual cortex and reinforcement learning the dopamine signal, while memory, bodies, energy and lifelong learning are where the two still differ most
T5 written from scratch and loaded with Google's weights: relative position buckets, cross-attention, span corruption and BART's text infilling, and a T5 fine-tuned on Spider to write SQL, checked by running every query on SQLite.
GPT-2 written from scratch and loaded with OpenAI's weights, generation with a KV cache, the logit lens, and a small GPT trained on Lichess games that learns to play legal chess and keeps a board inside, found with a linear probe.
How a search engine reads: BM25 written from scratch, Sentence-BERT's bi-encoder trained with in-batch negatives, a cross-encoder reranker and ColBERT's late interaction, each checked against the real weights and put together into a literature search over 64,000 machine-learning papers.
What seven years did to BERT: RoBERTa's training recipe, ELECTRA's replaced-token detection, DeBERTa's disentangled attention and ModernBERT's LLM-era block, each written in PyTorch and checked against the real weights, and the four compared on triaging GitHub issues of projects they never saw.
BERT written in PyTorch from scratch and loaded with Google's weights: why reading in both directions needed a mask, what BERT knows and what its attention heads do, and a detector of personal data in long financial documents, against regular expressions.
A mixture-of-experts layer in PyTorch trained next to dense models: the router, load balancing, dropped tokens, what experts learn, and serving
Linear attention, the delta rule and Gated DeltaNet in PyTorch, a recall test, GPU timings, and why Qwen3-Next and Kimi Linear keep few attention layers
Ship a model so that it runs the same next year: a lock file with hashes, an image pinned by digest, builds that give the same bytes twice, GitHub Actions publishing to GHCR, the image pulled and run without Docker, and the model exported to ONNX for a free in-browser demo.
Glaucon's challenge, Hobbes's Foole and Hume's farmers read through the prisoner's dilemma: when cooperation is rational, which virtues the models select, how much forgiveness reason allows, and what game theory cannot settle
Put a Transformer behind an HTTP API and measure it the way users meet it: a KV cache in beam search, int8 quantization that doesn't pay off, FastAPI and asyncio, open-loop load tests, queueing theory checked against the measurements, dynamic batching, and a Gradio page.
Turn a model from a notebook into a Python package anyone can install: the src layout, pyproject.toml, a command line, four kinds of tests checked by mutation testing, GitHub Actions, and a model card that measures what the model can't do.
In 1973 Allen Newell asked psychology for one system that performs all its tasks. In 2025 a Llama fine-tuned on 160 experiments was presented as that system. What Centaur predicts, what happens when you take away its instructions, and why its own authors went on to beat it with half a million parameters
Spend an hour of a free GPU on hyperparameters: experiment tracking with Weights & Biases, a learning-rate range test, random search and successive halving from scratch, and W&B Sweeps with Bayesian optimization.
Pre-norm, RMSNorm, SwiGLU and RoPE: the changes Llama made to the 2017 Transformer, written in PyTorch, added to our translator one at a time and measured against what a new random seed does alone, with fused attention and a KV cache.
Evaluate English–Russian translation with BLEU, chrF and COMET written from scratch: multiple references, agreement with professional translators, significance tests and a contaminated test set.
Train a Transformer on half a million sentence pairs on a free GPU, then decode it with greedy search, beam search and sampling.
Attention, masks, heads, positions and the full encoder-decoder of the 2017 paper, written line by line in PyTorch and checked against real models.
What GPT-4o actually receives when you type, why Russian costs more tokens than English, and byte-pair encoding written from scratch.
A patient who could reason about everything and decide nothing, a card game whose decks are rigged, and a theory that puts the body inside decisions: Damasio's somatic markers, the evidence for them and the evidence against
Why generating text is limited by memory rather than arithmetic, measured on a free GPU, and what serving engines like vLLM do about it: the KV cache, continuous batching, quantization and speculative decoding
How to adapt a pretrained transformer by training a fraction of a percent of it: what adapters, LoRA, prefix and prompt tuning, BitFit, (IA)³ and Compacter add to the model, written out in PyTorch and measured on one task
Symbols, rules and meaning: the computational theory of mind from McCulloch and Pitts to the Chinese Room, and what language models add to an old argument
Positional encoding, self-attention, multi-head attention and cross-attention, then the encoder and decoder layers, written in PyTorch one module at a time.
Byte-pair encoding written in Python step by step, trained on English–Russian sentences from Tatoeba and compared with the tokenizers of Hugging Face.
What happens in a brain that has a new idea, where recombination stops, and what the measurements of machine creativity show, from AlphaGo's move 37 to GPT-4 on the Alternate Uses Task
Why a fine-tuning update can be the product of two thin matrices: rank, the singular value decomposition and the Eckart–Young theorem, then LoRA's initialization, gradients and scale, derived by hand and checked on BERT
What a large language model is, how it is trained in three stages, and what it could and could not do in spring 2024: a tour after Andrej Karpathy's talk, with the claims checked against their sources and measured on small open models
Naming, comments, modules, tests, docstrings, exceptions and keyword arguments: the basic habits that keep Python code readable and easy to change
