microgpt

In one sentence

The essence of a GPT model is not found in its massive scale, but in the fundamental recursive logic of predicting the next token using a chain-rule-based autograd engine.

Why read this (and who it's for)

This is for the engineer or researcher who feels like a "black box" user of AI. It solves the problem of intellectual dependency—where you can use a tool but cannot explain how it works. By stripping away the GPU-accelerated tensor libraries and the trillion-parameter scale, Karpathy allows you to see the "skeleton" of the machine. After this, you will be able to explain the flow of a gradient, the mechanics of a transformer block, and the logic of temperature-based sampling without relying on a library's abstraction.

The big idea

The central thesis is that the complexity of modern LLMs is a layer of "efficiency" added onto a relatively simple mathematical core. Karpathy argues that to master the field, one must first master the "bare essentials": a dataset, a tokenizer, an autograd engine, a model architecture, and a training loop.

The logic follows a "bottom-up" reconstruction. Instead of starting with a pre-built library like PyTorch, you start with a Value class that handles scalar differentiation. By building this, you realize that every operation in a neural network—from a simple multiplication to a complex multi-head attention mechanism—is just a sequence of operations where we need to know the "rate of change" (gradient) to update the weights.

Key ideas

The "Everything Else is Efficiency" Philosophy

• What it is: The belief that the core logic of LLMs can be captured in a ~200-line Python script without external dependencies.
• Why it matters: It removes the "magic" from AI. When you realize the core logic is just a loop of predicting the next character, the daunting scale of GPT-4 becomes a matter of engineering (data, compute, and parameters) rather than a mystery of "intelligence."
• In the book: The project is a culmination of a "decade-long obsession" to simplify LLMs to their bare essentials.

The Scalar Autograd Engine

• What it is: A manual implementation of a computation graph where each Value object stores its data, its gradient, and its children in the graph.
• Why it matters: It forces you to understand the Chain Rule. By manually handling _local_grads and accumulating them (+=) across branching paths, you see exactly how the "learning" in a model is mathematically propagated.
• In the book: The example of $L = a \cdot b + a$ where $a=2, b=3$ shows how $a.grad$ is calculated as $4.0$ (the sum of contributions from both paths).

Tokenization as Statistical Mapping

• What it is: The process of converting raw text into unique integer IDs (e.g., 'e' = 1, 'm' = 2).
• Why it matters: It defines the "universe" the model lives in. By using a 27-token vocabulary (26 letters + 1 BOS), the model learns to associate specific sequences of integers with the probability of the next integer.
• In the book: The model is trained on 32,000 names (e.g., "emma") to learn to "hallucinate" new, plausible-sounding names.

The Temperature of Creativity

• What it is: A parameter that divides the "logits" (raw scores) by a value before applying the softmax function.
• Why it matters: It allows for a controllable trade-off between predictability and variety. Low temperature (0.5) makes the model "conservative"; high temperature flattens the distribution for more diverse output.
• In the book: Temperature $\approx$ 0 is described as "greedy decoding," where the model always picks the single most likely next token.

The "Onion" Development Approach

• What it is: A pedagogical method of building a complex system by adding one layer of complexity at a time.
• Why it matters: It prevents cognitive overload. By moving from Bigram counts $\rightarrow$ MLP $\rightarrow$ Autograd $\rightarrow$ Attention $\rightarrow$ Adam Optimizer, you ensure each step is mastered before the next is added.
• In the book: The progression from train0.py to train.py represents this steady accumulation of complexity.

Frameworks & tools

The "Onion" Development Roadmap

To master LLM architecture, build the following in order:

1. Bigram Count Table: No neural net, no gradients. Just count occurrences of character pairs.
2. MLP + Manual Gradients: Implement a Multi-Layer Perceptron and calculate gradients manually (numerical and analytic).
3. Autograd Engine: Replace manual gradients with a Value class that handles the chain rule automatically.
4. Transformer Components: Add Position Embeddings, Single-head Attention, RMSNorm, and Residual connections.
5. Full Architecture: Integrate Multi-head Attention and the Layer loop.
6. Optimizer: Implement the Adam optimizer.
7. Final Integration: The full train.py script.

Temperature Control Checklist

When generating text, adjust the temperature based on your goal:

1. For Fact-Checking/Strict Logic: Set Temperature $\approx$ 0 (Greedy Decoding).
2. For Standard Interaction: Set Temperature = 1.0 (The model's learned distribution).
3. For Creative/Diverse Writing: Set High Temperature (Flattens the distribution).
4. For Focused/Conservative Output: Set Low Temperature (e.g., 0.5).

Memorable examples & stories

The Name Hallucination

The model is trained on a small list of 32,000 names. The goal is not to memorize the list, but for the model to learn the statistical patterns of what a name looks like so it can "hallucinate" new ones. Lesson: Training is about capturing patterns, not just memorizing a database.

The $L = a \cdot b + a$ Gradient Example

To explain how gradients accumulate, Karpathy uses a simple equation where $a=2$ and $b=3$. Because $a$ appears twice in the equation, its gradient is the sum of the contributions from both parts of the expression. Lesson: Gradients are additive; when a value branches in a graph, the total derivative is the sum of all paths.

The Production vs. Micro Scale

Karpathy contrasts MicroGPT (4,192 parameters) with GPT-4 (hundreds of billions). While the mechanics are the same, the scale of data (trillions of tokens) and hardware (TPUs/GPUs) is what creates the "intelligence" we see today. Lesson: Architecture provides the capability; scale provides the power.

Key quotes

"Everything else is just efficiency." Why it matters: This reinforces that the core logic of a transformer is independent of the massive engineering required to make it run at scale.

"The statistical patterns of the training sequences are distilled into the model parameters." (Paraphrased) Why it matters: This defines what "training" actually is—a compression of probability distributions into weights.

"Lower loss indicates better prediction of the next token in a sequence." (Paraphrased) Why it matters: It provides a clear, mathematical metric for "learning" in a generative context.

How to apply it

This week

• Audit your "Black Box" tools: Identify one AI tool or library you use daily. Write down the 3 core mathematical principles it relies on (e.g., "It uses a gradient-based optimizer to minimize a loss function").
• Trace a single token: Take a simple sentence. Manually write out the "next token" probability for each word. What patterns do you notice?

Ongoing

• The "Explain it to a child" Rule: Whenever you learn a new LLM concept (like RoPE or MoE), try to explain how it would work if you only had a Value class and a for loop.
• Build, don't just watch: When learning a new paper, try to implement the smallest possible version of the core idea (the "Micro" version) before looking at the full production code.

Common pitfalls

• The Scale Fallacy: Assuming that because a model is "smart," it must be using a fundamentally different logic than a tiny model. It isn't; it's just a much larger version of the same math.
• Ignoring the Autograd: Many beginners try to learn Transformers without understanding how gradients flow. Without understanding the Value class/Chain Rule, the "Attention" mechanism remains a magic trick.
• Over-complicating the Tokenizer: Thinking you need a complex BPE tokenizer to start. For learning, a simple character-level mapping is often more transparent.

Connects to

• Deep Learning (Goodfellow et al.): Provides the formal mathematical backing for the gradients and optimization Karpathy implements.
• The "Build-to-Learn" Philosophy: Similar to the "learning by doing" approach in coding, this challenges the "passive consumption" of AI theory.

If you remember only three things

• The core of an LLM is a predictive loop: Given a sequence, what is the most likely next token?
• Autograd is the engine: All learning happens because we can calculate the "rate of change" (gradient) of the loss relative to every single parameter in the network.
• Complexity is an Onion: You can understand the most complex AI in the world by starting with a simple Bigram count and adding layers of complexity one by one.

Internalize it

• Reflection questions

  • If I had to explain how a model "learns" without using the word "intelligence," how would I describe the gradient update?
  • How does the "Onion" approach of learning apply to other complex skills I am trying to master?
  • Why is it important to understand the "Micro" version of a technology before trying to master the "Production" version?

• Flashcards Q: What is the primary purpose of the Value class in MicroGPT? A: To provide a scalar-based autograd engine that tracks data, gradients, and the computation graph. Q: Why do we use += when accumulating gradients? A: Because if a value