How large language models work, for those who missed out

In 2023 the lawyers of a man suing the airline Avianca filed a brief in a New York federal court. It cited earlier decisions that the other side could not find, and neither could the judge.

One of the lawyers, Steven Schwartz, had found them with ChatGPT, and had even checked: “Is Varghese a real case?” ChatGPT replied that the case “does indeed exist” and could be found “on Westlaw and LexisNexis”.

It did not exist. In June the court fined the lawyers and their firm $5,000, and the judge called the legal analysis in one of the invented opinions “gibberish”^[1].

The judge was careful to add that “there is nothing inherently improper about using a reliable artificial intelligence tool for assistance”. The trouble was what the lawyers took the tool to be: a database of court decisions. It is not one.

This post is about what a large language model actually is. It follows a talk Andrej Karpathy recorded in November 2023 for people who had not been following the field, “Intro to Large Language Models” (video, slides).

The route is the same: what a model is made of, the three stages of training, what models can do, how they fail. Along the way I check the claims against their sources and measure some of them on small open models that run on a laptop.

Two files

Karpathy starts with a surprise: a language model you can download is two files. One holds the parameters. For Llama 2 70B, which Meta released in July 2023 with its weights open, that is 70 billion numbers, each stored in two bytes: 140 GB.

The other file is a program that reads those numbers and uses them to compute what text should come next. It is short.

Karpathy’s own llama2.c began in July 2023 as a single C file of 492 lines with no dependencies; the version current today has 973. (It also computes in 32 bits, and its README warns that models larger than 7B will not load productively.)

Running the model needs no internet, only enough memory to hold the parameters, and that is the catch. 140 GB is more than any laptop has: the largest MacBook Pro tops out at 128 GB^[2]. Karpathy’s demo in the talk was the 7B model, and Karpathy notes that 70B would run “about 10 times slower”.

What makes 70B fit is rounding. Store every parameter in 4 bits instead of 16, in the format of the open-source program llama.cpp, and the model shrinks to 39 GB (GGUF files). That fits in a 64 GB MacBook Pro, at a small cost in accuracy.

A note on words: Llama 2 is not open source in the usual sense. Its license forbids, for instance, using its outputs to improve other models, and the Open Source Initiative says it is “very plainly not an ‘Open Source’ license”^[3]. “Open weights” is the accurate term.

Stage one: pretraining

The parameters are cheap to run and expensive to make. Karpathy’s slide gives round numbers for Llama 2 70B: about 10 TB of text, about 6,000 GPUs, about 12 days, about two million dollars.

These are Karpathy’s estimates. Meta’s paper gives the measured version: 2 trillion tokens of training text and 1,720,320 hours of A100 GPU time for the 70B model^[4]. Six thousand GPUs for twelve days is 1.73 million GPU-hours, so the round numbers hold up.

And Karpathy called them “rookie numbers”. Llama 3, released last week, was trained on “over 15T tokens”, seven times as much, and its 70B model took 6.4 million hours on newer H100 GPUs^[5]. The closed models are bigger still, by estimates only, since their makers do not publish them:

One question, asked again and again

What does all that computation teach the model? One skill: given a piece of text, assign a probability to every possible next token, a word or a piece of a word. Here is a small open model, TinyLlama, with 1.1 billion parameters trained on about a trillion tokens three times over^[6]:

“Can” gets 48%, and it is only the first of three pieces: the model reads and writes “Canberra” as “Can”, “ber”, “ra”. “Sydney” gets 10%. That is the popular mistake, and it lives in the model as a probability, learned from all the text that gets the capital wrong.

To write, the program asks the same question again and again. It picks a token, appends it to the text, and asks for the next one:

Always taking the most likely token, the model finishes the sentence, starts a new line and writes the same sentence again, more sure of itself the second time. Repetition is a well-known failure of this “greedy” choice.

Chatbots instead draw each token at random, in proportion to its probability, which is also why the same question can get different answers, and why one answer in ten might say Sydney.

Prediction is compression

Karpathy describes pretraining as compressing the internet into the parameters, “kind of like a zip file”. The comparison is more literal than it sounds.

A model that gives probability \(p\) to the token that actually comes next can encode that token in \(-\log_2 p\) bits, with a method called arithmetic coding, and decode it back exactly.

The better the predictions, the shorter the code. Grégoire Delétang and colleagues at Google DeepMind took this seriously: Chinchilla, a 70-billion-parameter model, compresses a gigabyte of Wikipedia to 8.3% of its size, where gzip gets 32.3%^[7].

I tried the same on a text none of these models could have seen: an earlier draft of this post, written this month.

On this unseen text, TinyLlama needs 0.86 bits per character, about a quarter of gzip’s 3.65.

GPT-2, from 2019, needs 1.20. The Pythia models were all trained on the same data in the same order^[8], so they differ only in size, and every step up in size gives a shorter code.

There is a catch the chart leaves out: to decode, you need the model, and TinyLlama’s parameters take 2.2 GB, three hundred thousand times more than the text.

As a practical compressor a language model is absurd; Delétang et al. count the parameters in and get rates in the thousands of percent. As a measure of how much a model has learned about text, the code length is exact.

Lossy

Arithmetic coding with a model is lossless. What is lossy is the model itself as a summary of its training data: 2 trillion tokens do not fit into 140 GB. Karpathy puts the ratio at roughly a hundred to one.

What the model keeps is closer to the gist of the data than to the data, and when asked for details it does not have, it produces text with the shape of the details:

TinyLlama knows the form of the answer, a name, and it knows that Tom Cruise and Nicole Kidman go together. Large models make the same kind of error less often, but they make it, as the lawyers in New York found out.

There is a stranger failure. Karpathy’s example: GPT-4 knows that Tom Cruise’s mother is Mary Lee Pfeiffer, but asked who Mary Lee Pfeiffer’s son is, it says it does not know. Lukas Berglund and colleagues called this the reversal curse and measured it on celebrity parents: GPT-4 named the parent 79% of the time and the child 33%^[9].

Why would a model know a fact one way round and not the other? It learned the fact from text in which Tom Cruise comes first and Mary Lee Pfeiffer second, and training on “A is B” makes B more likely after A, not A after B. Berglund et al. tested this directly.

They fine-tuned GPT-3 and Llama-1 on invented facts such as “Uriah Hawthorne is the composer of Abyssal Melodies”, and the models could not answer “Who composed Abyssal Melodies?”: the right name was no more likely than a random one.

Stage two: from a document completer to an assistant

A model fresh from pretraining continues documents. Asked a question, it keeps writing the page the question might have come from:

The base model repeats the question, answers it, and repeats the pair, the way a page of quiz questions would. The chat version answers, once. The difference comes from a second, much smaller round of training on conversations written by people, in which the model learns the format and tone of an assistant.

OpenAI’s InstructGPT, the predecessor of ChatGPT, was trained this way on about 13,000 prompts with answers written by “a team of 40 contractors”. Labellers preferred the answers of the 1.3-billion-parameter result to those of the original 175-billion-parameter GPT-3^[10].

Meta stopped at 27,540 annotated examples for Llama 2, under a heading that says it all: “Quality Is All You Need”. Karpathy’s slide says “~100K conversations”; for Llama 3, Meta reports “over 10M human-annotated examples” across all of its fine-tuning.

The bottom row of the figure shows what the chat model actually receives. The question is wrapped in markers, <|user|> and <|assistant|>, and the model completes the document as before.

Fine-tuning did not change what the model does, only which documents it expects. Because this stage is cheap, Karpathy notes that companies can repeat it “every week”, while pretraining happens perhaps once a year.

Stage three: comparing instead of writing

It is easier to judge a haiku than to write one. The Llama 2 paper puts it this way: “while we may not all be accomplished artists, our ability to appreciate and critique art remains intact”. (TinyLlama’s haiku above is not 5-7-5, and anyone can see it.)

So the third stage collects judgments. People are shown two answers to the same prompt and pick the better one. Meta collected 1,418,091 such comparisons for Llama 2.

A separate reward model is trained to predict the choices, and the chat model is then trained, with reinforcement learning, to write answers the reward model scores highly.

The method is reinforcement learning from human feedback (RLHF). OpenAI made it famous with InstructGPT, but it goes back to a 2017 paper by researchers from OpenAI and DeepMind together^[11].

In 2023 a shortcut appeared, direct preference optimization (DPO), which learns from the same comparisons without a reward model^[12]. TinyLlama’s chat version was trained with it, on comparisons ranked by GPT-4 instead of people (model card).

Who is ahead

How do you rank chatbots? The most trusted answer in 2024 is a tournament. On Chatbot Arena, run by the LMSYS group at Berkeley, a visitor asks a question, two anonymous models answer, and the visitor votes for the better one.

From the votes, more than 240,000 by January, the site fits a rating like a chess player’s Elo^[13].

In November 2023 Karpathy’s slide showed the closed models, GPT-4 and Claude, well above every model with open weights.

In late March Claude 3 Opus became the first model to share the top spot with GPT-4. Then, last Thursday, Meta released Llama 3. This is the board on Monday:

Llama 3 70B, whose weights anyone can download, sits fifth: below the GPT-4 Turbo versions and Claude 3 Opus, above Claude 3 Sonnet and the original GPT-4 of March 2023.

Llama 2 70B, the open model of Karpathy’s talk, is at the bottom of this list. The gap between closed and open has not closed, but in a year it shrank from a wide margin to a few dozen rating points.

The choice between them is still the one Karpathy described. Closed models are rented through a website or an API.

Some can be fine-tuned through the API (OpenAI has offered this for GPT-3.5 since August 2023), and in November OpenAI added custom “GPTs”, ChatGPT with your own instructions and files, followed in January by a store for them. Open weights can be downloaded, run privately, and changed without asking anyone.

Tools and senses

A language model on its own is bad at arithmetic and knows nothing after its training data ends. Both problems have the same fix: let it call programs. The model writes a request in an agreed format, the program around it runs the tool, and the result is pasted back into the text for the model to read:

This is the lesson of the reversal curse put to work: what the model cannot recall from training, it can still use when the text in front of it contains it.

Researchers showed in early 2023 that a model can teach itself when to call a calculator or a search engine^[14]. In March 2023 OpenAI opened ChatGPT to plugins, later replaced by GPTs; plugins were switched off this month.

The tools that remain built in are web search through Bing, a Python sandbox for analysis and charts, and DALL·E 3 for images.

Karpathy’s demo strings them together: it searches for a startup’s funding rounds, computes the missing valuations with a calculator, plots them with Python, extrapolates the trend, and draws a logo.

The senses arrived in the same year:

• Images in. Since September 2023 ChatGPT can look at photos^[15], and since March 2024 so can Claude 3^[16].
• Voice. ChatGPT talks, but through a pipeline: a speech recognizer, Whisper, writes down what you say, and a separate speech synthesizer reads the answer aloud.
• Long inputs. Google’s Gemini 1.5 Pro, in testing since February, reads up to a million tokens: “1 hour of video, 11 hours of audio” or a stack of books^[17].
• Video out. Not yet. OpenAI showed Sora in February, a model that generates minute-long videos, but has given it only to testers and a few artists^[18].

Thinking slowly

Daniel Kahneman, who died last month, described the mind with two characters. “System 1 operates automatically and quickly, with little or no effort.” System 2 takes over for “effortful mental activities”, such as the book’s example 17 × 24: you know at once that it is a multiplication, but the answer, 408, takes work^[19].

A language model, Karpathy says, has only a System 1. Every token costs the same fixed amount of computation, whether the question is “2 + 2” or a proof. The model cannot stop and think before it writes; it can only write.

But writing can stand in for thinking. Ask a model to reason step by step, and the intermediate steps become part of the text it reads before answering. Adding one sentence, “Let’s think step by step”, raised an OpenAI model’s score on simple arithmetic word problems from 17.7% to 78.7%^[20].

The next step is to let the model try several lines of reasoning, judge them, and back up from dead ends, as a person does with a hard puzzle. Shunyu Yao and colleagues called this a tree of thoughts and tested it on the Game of 24:

Searching through a tree of partial solutions, judging each with the model itself, GPT-4 solved 74% of the puzzles, against 7% answering at once and 4% reasoning step by step^[21].

The search costs far more computation per answer. Karpathy’s wish is a model you can tell to take thirty minutes and come back with a better answer; no product offers that today.

Can a model teach itself?

In March 2016 DeepMind’s AlphaGo beat Lee Sedol, one of the strongest Go players in the world, 4–1. It learned in two stages.

First it imitated people: a network trained on 30 million positions from online games learned to predict the expert’s move 57% of the time. Then it played against itself, and the self-taught version won more than 80% of its games against the imitator^[22].

A year later DeepMind dropped the first stage. AlphaGo Zero started from random play, learned only from games against itself, and after three days beat the version that had beaten Lee Sedol 100 games to 0^[23]:

Language models, Karpathy says, are still in AlphaGo’s first stage: they imitate people. The obstacle to the second is the “lack of a reward criterion in the general case”.

Go has an exact one, the game is won or lost. “Was this a good essay?” has none.

Where a check exists, the second stage has begun:

• Geometry. AlphaGeometry trained on 100 million synthetic theorems with proofs it generated itself, and solved 25 of 30 olympiad geometry problems^[24].
• Mathematics by program. FunSearch paired a language model with a program that scores its proposals and found larger “cap sets”, a problem in combinatorics, than mathematicians had^[25].
• Models as judges. Anthropic’s Constitutional AI replaces human judgments of harmlessness with judgments by a model following written principles^[26]. In January Meta reported a Llama 2 70B that improves by grading its own answers^[27].

Attacks

The last part of Karpathy’s talk is about security, and it is the part most people skip. A model that follows instructions in text will follow instructions from anyone who can put text in front of it.

• Jailbreaks. Safety training is a habit, not a lock. In April 2023 people got Discord’s chatbot to explain how to make napalm by asking it to play their late grandmother, who used to tell them about her work at a napalm factory^[28]. The same request encoded in Base64 slips past refusals that were trained mostly on plain English^[29].
• Adversarial suffixes. A string of nonsense characters, found by optimizing against open models, makes closed ones comply too; in the paper’s tests it worked on GPT-3.5 86.6% of the time^[30]. A slightly altered image of a panda does the same to a model that reads images^[31].
• Prompt injection. A web page, an email or a shared document can contain instructions aimed at the model that reads it. Such applications “blur the line between data and instructions”^[32]; one researcher got Google’s Bard to leak a user’s data through a shared Google Doc^[33].
• Poisoned data. As few as 100 poisoned examples in fine-tuning data can make a model misbehave whenever a trigger phrase, “James Bond” in the paper’s example, appears^[34]. In January Anthropic showed that such hidden behaviour can survive standard safety training^[35].
• Long contexts. And three weeks ago Anthropic described “many-shot jailbreaking”: fill a long context window with hundreds of fake dialogues in which an assistant complies, and the real one starts to comply as well^[36].

What comes next

Karpathy ends with a picture: the language model as the kernel of a new operating system. The context window is its working memory, the tools are its devices, and the model decides what to load and what to run. The talk’s list of what such a system will soon do reads, five months later, like this:

• read and write text, browse the internet and use existing software: done, through tools;
• see and generate images: done; generate video: shown, not released;
• hear and speak: done, through a pipeline of separate models;
• think for a long time before answering: not yet, beyond research such as Tree of Thoughts;
• improve itself where success can be checked: in research, in geometry and mathematics;
• be customized and sold in an app store: done, as GPTs;
• talk to other models: in research frameworks.

The other half of the operating-system picture is the market. On one side are the closed systems, like Windows and macOS; on the other, open weights, like Linux, now a few dozen rating points behind.

Whichever wins, the lawyers’ lesson holds. What is on the other side of the chat window is not a database. It is a very good predictor of the next word, trained on most of what people have written.

References

1. Mata v. Avianca, 2023. storage.courtlistener.com. ↩
2. Apple, 2023. apple.com. ↩
3. OSI, 2023. opensource.org. ↩
4. Touvron et al., 2023. arxiv.org. ↩
5. Meta, 2024. ai.meta.com. ↩
6. Zhang et al., 2024. arxiv.org. ↩
7. Delétang et al., 2023. arxiv.org. ↩
8. Biderman et al., 2023. arxiv.org. ↩
9. Berglund et al., 2023. arxiv.org. ↩
10. Ouyang et al., 2022. arxiv.org. ↩
11. Christiano et al., 2017. arxiv.org. ↩
12. Rafailov et al., 2023. arxiv.org. ↩
13. Chiang et al., 2024. arxiv.org. ↩
14. Schick et al., 2023. arxiv.org. ↩
15. OpenAI, 2023. openai.com. ↩
16. Anthropic, 2024. anthropic.com. ↩
17. Google, 2024. blog.google. ↩
18. OpenAI, 2024. openai.com. ↩
19. Kahneman, 2011. scientificamerican.com. ↩
20. Kojima et al., 2022. arxiv.org. ↩
21. Yao et al., 2023. arxiv.org. ↩
22. Silver et al., 2016. doi.org. ↩
23. Silver et al., 2017. doi.org. ↩
24. Trinh et al., 2024. doi.org. ↩
25. Romera-Paredes et al., 2023. doi.org. ↩
26. Bai et al., 2022. arxiv.org. ↩
27. Yuan et al., 2024. arxiv.org. ↩
28. Kotaku, 2023. kotaku.com. ↩
29. Wei et al., 2023. arxiv.org. ↩
30. Zou et al., 2023. arxiv.org. ↩
31. Qi et al., 2023. arxiv.org. ↩
32. Greshake et al., 2023. arxiv.org. ↩
33. Embrace The Red, 2023. embracethered.com. ↩
34. Wan et al., 2023. arxiv.org. ↩
35. Hubinger et al., 2024. arxiv.org. ↩
36. Anthropic, 2024. anthropic.com. ↩

@misc{koroleva2024llmsforthosewhomissedout,
  author       = {Koroleva, Eva},
  title        = {{How large language models work, for those who missed out}},
  year         = {2024},
  month        = dec,
  howpublished = {Eva Koroleva's blog},
  url          = {https://blog.korolevalab.com/blog/2024/llms-for-those-who-missed-out/}
}