Deep Reinforcement Learning Pong from Pixels

In one sentence

The core of modern reinforcement learning (RL) is a "fancy form of guess-and-check" where an agent samples actions (the guess), observes the outcome, and uses policy gradients to nudge the probability distribution toward higher rewards (the check).

Why read this (and who it's for)

This is for engineers and researchers who want to move past treating RL as a "black box." It explains the mechanics of how an agent learns from raw pixels, why certain algorithms (like Policy Gradients) are preferred over others, and the fundamental difference between how humans learn via abstraction versus how AI learns via brute force. You will be able to identify why some environments are "hard" for RL and how to architect systems that handle non-differentiable components like hard attention or memory I/O.

The big idea

The central thesis is that progress in AI is not primarily driven by "new amazing ideas," but rather by the scaling of four pillars: Compute, Data, Algorithms, and Infrastructure. For example, AlexNet was essentially a scaled-up version of 1990s ConvNets; similarly, modern RL success comes from applying standard components (like CNNs as function approximators) to massive amounts of data and compute.

The logic of the "Pong" experiment demonstrates that we can train an agent to play a game using only raw pixels and a binary reward signal (+1 for win, -1 for loss). By treating the policy as a stochastic distribution, we use the Score Function Gradient Estimator. This allows us to nudge the parameters ($\theta$) in the direction of $\nabla_{\theta} \log p(x)$ multiplied by the scalar score $f(x)$. In plain English: if an action led to a win, we increase its probability; if it led to a loss, we decrease it.

Key ideas

Policy Gradients (PG) vs. Q-Learning

• What it is — PG is an end-to-end approach that directly optimizes the expected reward by maintaining an explicit policy distribution.
• Why it matters — It is the preferred default because it handles stochastic policies and avoids some of the pitfalls of Q-learning (which Karpathy notes is "so 2013").
• In the book — The author uses a 2-layer fully-connected neural network to map $100,800$ input numbers directly to the probability of moving UP.

The Credit Assignment Problem

• What it is — The difficulty of determining which specific action in a long sequence (e.g., an action taken 76 frames ago) actually caused a final reward.
• Why it matters — Without solving this, the agent cannot learn which behaviors are responsible for success.
• In the book — This is addressed by running "rollouts" and labeling every decision in that sequence based on the final outcome (the Advantage $A$).

Difference Frames

• What it is — A preprocessing technique where you feed the network the difference between the current frame and the previous frame ($current - last$).
• Why it matters — It allows the network to detect motion/velocity, which is crucial for predicting ball trajectories.
• In the book — The Pong agent uses $80 \times 80$ difference images as input rather than raw static frames.

Human vs. AI Learning (Abstraction vs. Brute Force)

• What it is — Humans learn via "rapid abstract model building" (physics, psychology, goals); RL agents learn via "brute force" (stumbling into rewards).
• Why it matters — This explains why RL struggles with games like Montezuma's Revenge where rewards are hard to "stumble into."
• In the book — The author contrasts how a human understands a slow opponent allows for high vertical velocity, while an agent must experience that transition hundreds of times to internalize it.

Non-differentiable Computation

• What it is — Many operations (like sampling a location in memory) are non-differentiable because the outcome of different samples is unknown during the forward pass.
• Why it matters — It limits how we can train complex architectures like Neural Turing Machines or hard attention mechanisms.
• In the book — The solution is to treat the sampling as a stochastic policy and use Policy Gradients to update parameters based on the success of those samples.

Frameworks & tools

The Policy Gradient Loss Function

To train an agent, substitute "fake" labels (the action actually sampled) into a supervised learning loss, modulated by an Advantage ($A$):

1. Run Rollouts: Execute 100 games/episodes.
2. Calculate Advantage: Determine $A$ (e.g., $+1$ for win, $-1$ for loss).
3. Compute Loss: $\sum A_i \log p(y_i \mid x_i)$.
4. Backpropagate: Update parameters to increase the probability of actions with positive $A$.

Trust Region Policy Optimization (TRPO)

Use this variation of PG for more consistent results:

1. Constraint: Ensure the new policy distribution doesn't deviate too far from the old one.
2. Mechanism: Use KL divergence constraints to avoid large, destructive parameter updates.

The "BB Gun" (Cross-Entropy Method)

Before reaching for the "Bazooka" of Policy Gradients:

1. Method: Use CEM (stochastic hill-climbing).
2. Application: Try this first on simpler tasks where exploration is easier.

Memorable examples & stories

The Pong Agent Strategy

The agent learns to wait for the ball, then rapidly dashes to catch it at the edge of the screen to launch it with high vertical velocity. Lesson: Brute force discovery can lead to highly effective, non-intuitive strategies.

Montezuma's Revenge

This game is notoriously difficult for RL because the agent cannot easily "stumble" into rewards like jumping or climbing. Lesson: Without prior knowledge or dense rewards, brute force search fails.

AlphaGo’s Pipeline

AlphaGo uses supervised learning first (learning from human experts) before switching to Policy Gradients. Lesson: Combining supervision with RL can overcome the "stumbling" problem of pure exploration.

Key quotes

"The progress in RL is not driven as much as you might reasonably assume by new amazing ideas." Why it matters: It highlights that scaling compute/data is currently more effective than seeking a "magic" algorithm.

"At the core the approach we use is also really quite profoundly dumb." Why it matters: It strips away the mysticism of AI, revealing it as a simple gradient-based nudge of probabilities.

"Policy Gradients are a brute force solution, where the correct actions are eventually discovered and internalized into a policy." Why it matters: It defines the fundamental limitation of current RL—it requires frequent positive rewards to learn.

How to apply it

This week

• Difference Frames: If training an agent on video/images, subtract the previous frame from the current one to highlight motion.
• Baseline Check: Before implementing a complex Policy Gradient, see if the Cross-Entropy Method (CEM) can solve the task first.

Ongoing

• Prioritize Scaling: When stuck, look for ways to increase data quality or compute before trying to invent a new architecture.
• Identify "Stumbling" Barriers: When an agent fails to learn, ask: "Is it because the reward is too sparse to stumble into?" If so, consider adding supervised learning (expert trajectories).

Common pitfalls

• Thinking RL is Magic: Expecting the agent to "understand" physics. It doesn't; it just learns a mapping of pixels to actions via brute force.
• Ignoring Credit Assignment: Trying to train an agent without a clear way to link late-game rewards to early-game actions.
• Overcomplicating Early: Reaching for Policy Gradients (the Bazooka) before trying simpler stochastic hill-climbing (the BB gun).

Connects to

• AlphaGo: Extends the idea by showing how supervised learning can "seed" a policy so it doesn't have to start from scratch.
• Neural Turing Machines: Uses PG to solve the non-differentiable problem of hard memory I/O.
• Robotics Scaling: Challenges the brute force approach, suggesting that trajectory optimization or apprenticeship learning may be necessary for real-world robots.

If you remember only three things

• Progress is driven by scaling (Compute/Data/Infrastructure) rather than "magic" ideas.
• Policy Gradients work by treating actions as a distribution and nudging them based on the Advantage ($A$).
• AI learns via brute force; humans learn via abstraction.

Internalize it

• Reflection questions

  • In my current project, am I trying to find a "magic" algorithm solution when I actually need more data or compute?
  • Is the environment I'm modeling too sparse for an agent to "stumble" into rewards? How can I provide a "hint" (supervision)?
  • Can I simplify my input by using difference frames to highlight what actually matters (motion)?

• Flashcards Q: What are the four primary drivers of AI progress? A: Compute, Data, Algorithms, and Infrastructure. Q: Why is Policy Gradient preferred over Q-Learning in this context? A: It is end-to-end, has an explicit policy, and directly optimizes expected reward. Q: What is the "Credit Assignment Problem"? A: The difficulty of determining which specific action in a long sequence caused a final reward. Q: How does the agent "see" motion in the Pong experiment? A: By using difference frames (current frame minus last frame). Q: Why is sampling non-differentiable? A: Because the outcome of different samples is unknown during the forward pass. Q: What is the "BB gun" recommendation? A: Try the Cross-entropy method (CEM) first before using Policy Gradients.

Fidelity note

Confidence: High. This summary is grounded strictly in the provided notes. I am certain about the distinction between human/AI learning, the mechanics of Policy Gradients, and the specific Pong experiment details. Note that "Pong from Pixels" refers to a blog post/experiment by Karpathy rather than a formal textbook.