Eric Jang — Building AlphaGo from scratch

In one sentence

The core breakthrough of AlphaGo is using deep learning to truncate the depth and breadth of a massive game tree by mimicking the human ability to estimate win probability without playing out every possible move.

Why read this (and who it's for)

This summary is for researchers, engineers, and AI enthusiasts who want to understand the mechanics of how AlphaGo actually works—not just the "magic" of its success. It solves the problem of understanding how to handle high-dimensional search problems where the branching factor is too large for brute force. After reading this, you will be able to distinguish between MCTS and naive RL, understand the concept of "amortizing" simulation, and identify when to use macrostructure-focused neural networks versus granularer models.

The big idea

The fundamental problem with Go is its combinatorial complexity. With a branching factor of $\approx 361$ moves per turn over $\approx 300$ total moves, the number of possible outcomes exceeds the number of atoms in the universe. Classic AI faces two hurdles here: Breadth (too many actions to estimate individually) and Depth (hard to know which actions to sample without playing them to the end).

AlphaGo solves this by "amortizing" the simulation of something so deep in the game tree. Instead of brute-forcing every branch, it uses a Value Network to provide an analog for human intuition. Humans possess an implicit neural network that allows them to evaluate win probability from a board state at a glance without needing to play until the terminal leaf nodes. AlphaGo mimics this by training a value function to truncate search depth, allowing the system to "quickly resolve the game without playing out all of these trees into a very deep search depth."

The synthesis is MCTS (Monte Carlo Tree Search) aided by neural networks. MCTS provides an exhaustive search for every action taken, providing a high-fidelity supervision target with low variance. The neural network then provides the speed to make that search feasible. By combining these, AlphaG0 shrinks both breadth and depth simultaneously.

Key ideas

Amortizing Simulation

What it is — Using a Value Network to provide an "intuitive guess" ($V_\theta$) on the win probability of a board state. This allows the system to truncate search depth without playing out every logical tree. Why it matters — It turns an intractable search problem into a tractable one by replacing millions of simulations with a single forward pass through a network. In the book — The author notes that "the raw network actually takes all of the burden of that big TPU pod and just pushes it into the network."

MCTS as an Improvement Operator

What it is — MCTS refines an initial "diffuse" policy distribution (what we think might be good) into a "peakier" one by increasing simulations ($n$). Why it matters — It means you don't need a perfect starting point; the search process itself improves the model's understanding of which moves are best. In the book — The author explains that "the visit count that we store in the node earlier actually becomes the sort of vote for which way we should finally select an action here."

Distillation Strategy

What it is — Training the policy network to approximate the outcome of $N$ search steps. This allows the model to start from a better point by "amortizing" the first $N$ steps of search into the weights themselves. Why it matters — It prevents the model from wasting cycles on "garbage value predictions." You want to ensure the Value function is well-trained before investing heavy MCTS cycles. In the book — The author describes this as: "it's almost like if you could just, like, amortize the first 1,000 steps actually into the policy network instead of the search process."

Macrostructure vs. Microscale

What it is — Neural networks excel at capturing macrostructures (like the location of a hurricane) while ignoring chaotic micro-details (like wind velocity at a specific coordinate). Why it matters — Many problems are NP-hard in the worst case but possess internal structures that neural networks can exploit because real-world instances are rarely "worst-case." In the book — This is illustrated by the Lorenzo Tractor Analogy: one cannot predict exact future coordinates, but the overall "tractor" shape is predictable.

Supervision through a Straw vs. High-Fidelity Targets

What it is — Naive RL (like LLMs) reinforces actions from a winning trajectory, creating high variance because only one "true" signal exists among thousands of moves ("sucking supervision through a straw"). MCTS, conversely, provides a high-fidelity supervision target per move. Why it matters — High-fidelity targets lead to better sample efficiency and lower gradient variance in multi-step problems. In the book — The author notes that "the amount of information per flop has been decreasing" as trajectories get longer; MCTS avoids this by improving per move locally.

Frameworks & tools

The 4-Step MCTS Process

To run a search per move, follow these steps:

1. Selection: Choose the best action known so far based on selection criteria (e.g., UCB1: $argmax_a [Q(s, a) + \text{exploration bonus}]$).
2. Expansion: Grow the tree at a node not yet visited.
3. Evaluation: Run the node through the network to guess win/loss probability ($V_\theta$). (AlphaGo Lee Variation: $\alpha \cdot V_\theta + (1-\alpha) \cdot (\text{true randomly sampled playout})$).
4. Backup: Recursively update parent nodes with the mean Q-value (average across simulations).

NeuroFictitious SelfPlay

Use this for games where tree search is hard/non-deterministic (e.g., StarCraft):

1. Fix Opponent: Fix a strong opponent ($\Pi_B$).
2. Train Best Response: Use model-free RL (PPO, SAC) to train a policy ($\Pi_A$) specifically to beat $\Pi_B$.
3. Create League: Establish a league of various fixed opponents ($\Pi_B, \Pi_C, \dots$).
4. Distill Labels: Train best response policies for each opponent in the league.
5. Mixed Strategy: Distill these into a mixed strategy that averages across all possible opponents.

The "Daydreaming" Loop (Off-Policy Relabeling)

A method to minimize TD error using historical data:

1. Data Set: Collect trajectories $\rightarrow$ Reparser buffer pusher $\rightarrow$ Replay buffer.
2. Bellman Updater: Continuously "replans" best actions for historical states to minimize TD error.
3. Trainer: Fits $Q(s, a)$ to a $Q_{target}$.
4. Target Calculation: Compute $Q$ of $S'$, find action for $S'$ that maximizes $Q$, and add reward $\rightarrow$ actual target.

Memorable examples & stories

The Lorenzo Tractor Analogy: Used to illustrate chaos theory; while specific paths are unpredictable, the overall "shape" (macrostructure) is known. Lesson: Neural networks succeed by capturing macrostructures.

Lose the battle, win the war: A strategic dynamic in Go where a player allows an opponent to capture stones to gain a superior position for a larger capture elsewhere. Lesson: Value functions must understand long-term positioning over immediate local rewards.

AlphaFold: Used as an analogy for how AlphaGo works; it captures difficult physical simulations (protein folding) into a single network pass. Lesson: A small neural network can capture what feels like an MP-class problem into a single macroscopic simulation.

** The Bitter Lesson**: To study scaling laws, a system must be working first. Studying scaling on bad data or buggy systems yields unreliable results. Lesson: You cannot jump into the science of studying your artifact before it is interesting enough to be studied.

Key quotes

"Sucking supervision through a straw." Why it matters: Explains why naive RL on long trajectories is less sample-efficient than MCTS-based supervision.

"The bitter lesson." Why it matters: Warns against trying to derive scaling laws from buggy systems or bad data.

"Daydreaming." Why it matters: Describes the off-policy process of relabeling historical states with better actions in hindsight.

"Maximum power of a neural network exists at the 'edge of chaos'." Why it matters: Defines the structural sweet spot where networks can best approximate complex systems.

How to apply it

This week

• Identify your "Search" problem: Look for any process in your work that involves a massive branching factor (e.g., choosing between 100+ slightly different project paths).
• Look for Macrostructures: Instead of trying to predict the exact micro-scale outcome of a decision, identify the "Lorenzo Tractor"—the overall shape or direction you want to achieve.
• Audit your Supervision: Are you "sucking supervision through a straw" by only rewarding the final result? Can you find a way to provide higher-fidelity feedback for individual steps?

Ongoing

• Amortize your thinking: Whenever you find yourself doing a repetitive deep search, ask: "Can I train a 'Value Network' (a heuristic or checklist) to resolve this at a glance?"
• Prioritize Research Taste: Develop the ability to persevere through bugs based on a high-level belief in a "correct" idea. Top-down vision ensures stackable local improvements.

Common pitfalls

• Thinking algorithmic tricks are permanent: The author argues that many clever tricks (like those in the Katago paper) are merely transitory because they were needed for older hardware; as GPUs scale, these nuances matter less.
• Ignoring Value Grounding: Do not attempt to run MCTS on "garbage value predictions." You must ensure the value function is well-trained first via expert data or small-board transfer learning.
• Overestimating RL for complex tasks: If a problem has a clear tree search structure, MCTS is often more sample-efficient than trajectory reinforcement.

Connects to

• AlphaFold: Demonstrates how "simulation capture" works in biology.
• ** Scaling Laws (Kaplan/Chinchilla)**: Provides the benchmark for understanding compute growth.
• ** "As Rocks May Think"**: Explores the thesis of "thinking" as a primitive in computer science.

If you remember only three things

• AlphaGo succeeds by using neural networks to "amortize" the simulation of an intractable search tree.
• MCTS provides high-fidelity supervision per move, avoiding the low-signal problem of "supervision through a straw."
• Algorithmic heuristics are transitory; in the limit, compute and parameter scale are the primary determinants of intelligence.

Internalize it

• Reflection questions

  • Where in my current workflow am I performing an "intractable search" that could be amortized by a heuristic?
  • Am I focusing on micro-scale details (chaos) or macrostructures (the tractor)?
  • How can I provide higher-fidelity supervision for the individual steps of my project rather than just the final outcome?

• Flashcards Q: What does it mean to "amortize" a simulation in AlphaGo? A: Using a Value Network to estimate win probability, truncating the need to play out every branch of a game tree. Q: Why is MCTS more sample-efficient than naive RL for games like Go? A: Because it provides a high-fidelity supervision target for every action taken, rather than "sucking supervision through a straw" from a final winning trajectory. Q: What is the difference between macrostructure and microscale in neural networks? A: Networks excel at capturing macrostructures (e.g., hurricane locations) while ignoring chaotic micro-details (e.g., specific wind velocity). Q: What is "distillation" in the context of AlphaGo's training? A: Training the policy network to mimic the distribution of $N$ search steps, rather than just the single best action selected. Q: What is the "Lorenzo Tractor" analogy? A: A visual representation of chaotic systems where the overall "shape" is predictable even if specific paths are not.

Fidelity note

Confidence: High. This summary is grounded strictly in the provided notes from the Dwarkesh Podcast featuring Eric Jang. I am certain about the technical mechanics of MCTS, the Value/Policy network duality, and the philosophical arguments regarding compute vs. heuristics. The nuances of "research taste" and the specific "Daydreaming" relabeling loop are captured as described in the text.