Eric Jang — What rebuilding AlphaGo teaches us about self-play, RL, and future of LLMs

In one sentence

The breakthrough of AlphaGo lies in using neural networks to "amortize" the search process—using a value function to truncate the search tree and a policy network to narrow the branching factor—making intractable problems like Go solvable.

Why read this (and who it's for)

This summary is for AI researchers, engineers, and enthusiasts who want to move beyond "black box" understanding of LLMs and RL. It explains the mechanics of how high-level reasoning (search) interacts with low-level intuition (neural networks). You will be able to explain the MCTS/Neural Network synergy, understand the "distillation" of search into policy, and conceptualize how "test-time scaling" might apply to future LLM reasoning.

The big idea

The core thesis is that Go is a "tractable" problem only because we can use neural networks to represent human-like intuition. In a game with a branching factor of 361 and a depth of 300, the number of possible outcomes is larger than the number of atoms in the universe. We cannot search this tree exhaustively.

AlphaGo solves this by using two types of "intuition":

1. Policy Intuition: Instead of looking at every possible move, the AI uses a policy network to predict which moves are likely to be good, drastically narrowing the branching factor.
2. Value Intuition: Instead of playing every game to the very end (the "leaf" of the tree), the AI uses a value network to estimate the winner of a current board state, allowing it to "truncate" the search.

By combining these, the AI can focus its computational power only on the most promising branches of a much smaller tree.

Key ideas

The Curse of Dimensionality in Go

• What it is: The sheer scale of the Go game tree (361^300) makes it impossible to solve via brute-force search.
• Why it matters: It proves that "intelligence" in complex domains isn't just about raw compute; it's about the ability to prune the search space.
• In the book: Jang explains that while Go is deterministic, the "explosion" of possible outcomes requires a way to "merge" children (identifying that different paths can lead to the same state) to make the problem manageable.

MCTS and the PUCT Algorithm

• What it is: Monte Carlo Tree Search (MCTS) is a search algorithm that builds a tree of possible moves, while PUCT (Predicted Upper Confidence with Trees) is the specific scoring criterion used to select which branches to explore.
• Why it matters: It allows the AI to balance "exploration" (trying new moves) and "exploitation" (sticking to moves known to be good).
• In the book: Jang describes the data structure: each node stores a visit count, a mean action value ($Q$), and a probability ($P$) from the policy network. The algorithm selects the action that maximizes $Q$ plus an exploration bonus.

Policy vs. Value Networks

• What it is: The Policy Network predicts the "best" move at any given state; the Value Network predicts the probability of winning from that state.
• Why it matters: These two work in tandem to shrink the "breadth" and "depth" of the search tree simultaneously.
• In the book: Jang notes that while early versions (AlphaGo Lee) used two separate networks, modern versions merge them into two "heads" on a single network to share representations and save compute.

Distillation of Search into Policy

• What it is: The process of taking a model that has "thought" (used MCTS to search) and training a new model to mimic that thinking.
• Why it matters: This is the bridge to LLM reasoning. If you can "distill" 1,000 steps of search into a policy network, the model can "know" the answer without having to do the search every time.
• In the book: Jang explains that we can train a network to approximate the outcome of 1,000 MCTS steps, allowing the model to start at a much higher "win rate" before any search even begins.

The "Bitter Lesson" of Scaling

• What it is: The observation that scaling laws (increasing compute/data) often outperform clever algorithmic "tricks."
• Why it matters: It suggests that the path to AGI may involve finding the right "recipe" for scaling rather than just inventing a new rule for Go.
• In the book: Jang reflects on his attempt to build a Go bot using only scaling laws. He found that you often need a "working" system first to collect the data necessary to see the scaling laws clearly.

Frameworks & tools

The 4-Step MCTS Loop

To implement a search-based agent like AlphaGo, you must iterate through these four steps:

1. Selection: Use the PUCT criterion to choose the best action from the current tree.
2. Expansion: If the selected action leads to a state not yet in the tree, add it as a child node.
3. Simulation (Playout): Run a "rollout" (usually using the policy network) to get a single-sample estimate of the win/loss.
4. Backpropagation: Take the result of that simulation and update the $Q$ values of all parent nodes in the tree, moving from the leaf back to the root.

The DAgger-Style Correction

When training an agent to handle "noisy" environments (like a human opponent or a gust of wind in robotics):

1. Identify Optimal Trajectories: Define the "blue" states you want the agent to reach.
2. Identify Drift: Recognize that the agent will inevitably drift into "off-policy" states.
3. Funneling: Ensure the training data includes a "distribution of states" the agent might drift into, and provide a supervision signal that tells the agent how to return to the optimal trajectory.

Memorable examples & stories

The "Win the War, Lose the Battle" Principle

• The Story: In Go, it is often strategically sound to let an opponent capture a small group of stones if it allows you to secure a much larger territory elsewhere.
• The Lesson: Complex systems require a hierarchy of goals; local "losses" are irrelevant if they serve a global "win."

The "Sucking Supervision through a Straw" Analogy

• The Story: Referring to a comment by Andrej Karpathy, Jang discusses how LLMs are trained via policy gradient methods where the "signal" is incredibly sparse.
• The Lesson: In many RL problems, you spend 99.9% of your time in a "low pass rate" regime where you learn almost nothing. Success depends on how you initialize the model to ensure it has a non-zero chance of success.

The "Automated Scientist" Outer Loop

• The Story: Jang envisions using Go as a "verifiable" outer loop for an AI that is trying to discover new scientific laws.
• The Lesson: Because Go has clear win/loss conditions, it provides a perfect "grounding" for an AI to verify if its new ideas (in biology or physics) actually work without being "reward-hacked."

Key quotes

"AlphaGo's core conceptual breakthrough was using neural nets to make this search problem tractable." Why it matters: It shifts the focus from 'how do we search' to 'how do we make search easier.'

"You can think of it as... amortizing the first 1,000 steps into the policy network instead of the search process, then you could begin at a much better starting point." Why it matters: This is the fundamental logic behind "training" a model to think before it actually has to do the work.

"The amount you learn is a function of your pass rate... if you're just randomly guessing and seeing if it works, that's basically just the entropy of a binary random variable." Why it matters: Explains why RL is so hard—you learn the most from a coin flip (50% success), but you spend most of your time in the 0% success zone.

How to apply it

This week

• Analyze a "Search" problem: Identify a task in your work that feels like a "tree" (e.g., project planning, coding a complex feature). Ask: "What is the 'Policy' (the intuitive first guess) and what is the 'Value' (the ultimate success metric)?"
• Audit your "Pass Rate": Look at a learning goal you have. Are you spending too much time in the "low pass rate" zone (trying things that have 0% chance of success)? How can you "initialize" better?

Ongoing

• Think in "Distillations": When solving a hard problem, first do the "slow" version (the manual search/thinking). Then, try to "distill" that into a "fast" version (a checklist, a template, or a prompt) for next time.
• Verify via "Outer Loops": When experimenting with new ideas, find a "Go-like" verifiable metric (a win/loss) to ensure you aren't just optimizing for a proxy metric that doesn't actually solve the problem.

Common pitfalls

• The "Off-Policy" Trap: Beginners often think that training on "expert" data is enough. Jang warns that if you only train on the "perfect" path, the model will fail the moment it drifts slightly off-course. You must include "recovery" data.
• The "Search vs. Forward Pass" Confusion: Don't confuse a model's ability to predict a good move with its ability to search for one. They are different capabilities that can be traded off depending on how much compute you have at "test time."

Connects to

• The Bitter Lesson (Rich Sutton): How scaling and compute eventually trump human-designed heuristics.
• DAgger (Dataset Aggregation): The importance of training on the distribution of states the agent actually visits.
• System 1 vs. System 2 Thinking: The interaction between "fast" intuition (Policy) and "slow" deliberation (MCTS).

If you remember only three things

• Amortized Search: Neural networks don't replace search; they make search possible by narrowing the branching factor and truncating the depth.
• Distillation: You can "bake" the results of a slow, expensive search process into a fast, cheap policy network.
• The Value of Grounding: In complex RL, you need a "grounding" function (like a value network) to prevent the search from becoming untethered from reality.

Internalize it

• Reflection questions
  • If I were to "distill" my most complex professional skill into a "policy network," what would the "best" actions look like?
  • In my current projects, am I spending too much time in the "low pass rate" zone? How can I increase my "initialization" success?
  • How can I create a "verifiable outer loop" for my current research or learning goals to ensure I'm not "reward-hacking" my own progress?
• Flashcards Q: What are the two primary ways AlphaGo uses neural networks to make the game of Go tractable? A: It uses a Policy Network to narrow the branching factor (choosing which moves to consider) and a Value Network to truncate the search depth (estimating who is winning). Q: What is the core idea behind "distilling" search? A: It is the process of taking a model that has performed a deep search (like MCTS) and training a new model to mimic those results so it can provide the same answer without the search. Q: Why is the "low pass rate" a problem in Reinforcement Learning? A: Because if an agent has a 0% chance of success, it learns almost nothing; you learn the most when the success rate is near 50% (like a coin flip). Q: What is the purpose of the "Expansion" step in MCTS? A: To add new nodes to the tree when the agent encounters a state it hasn't explored before. Q: What is the "DAgger" perspective on off-policy training? A: You must train on the states the agent actually visits, including a "distribution of drift," so the agent learns how to recover from mistakes. Q: How does AlphaGo handle the "Branching Factor" of Go? A: It uses a Policy Network to predict which moves are good, allowing it to ignore the vast majority of possible moves at any given turn. Q: What is the "Outer Loop" in the context of automated AI research? A: A verifiable environment (like a game of Go) where an AI can test and verify the success of new algorithmic ideas.

Fidelity note

This summary is based entirely on the transcript provided. I am highly confident in the technical descriptions of MCTS, PUCT, and the Policy/Value split. The section on "Distillation" and "Test-time scaling" is a conceptual bridge drawn from Jang's explanation of how search can be "amortized" into a forward pass. The "DAgger" and "Bitter Lesson" references are contextualized by Jang's specific mentions of those concepts during the conversation.