George Hotz — Petaflops to the People

In one sentence

The core of modern ML infrastructure is over-engineered; we must move toward a "Reduced Instruction Set" (RISC) for machine learning to achieve true performance and accessibility.

Why read this (and who it's for)

This conversation is for engineers, researchers, and hardware enthusiasts who feel the current ML stack (PyTorch, XLA, etc.) is becoming a bloated, opaque mess. It explains why "completeness" in software design is often a liability and provides a roadmap for how to build a leaner, more performant ML stack from the ground up. You will learn how to think about ML as a static, predictable computation rather than a dynamic, branching nightmare.

The big idea

George Hotz argues that the current state of machine learning frameworks is analogous to the early days of computer architecture: we have "Complex Instruction Set" (CISC) systems that are bloated and difficult to optimize. He posits that because neural networks are fundamentally static—meaning the sequence of loads, stores, and arithmetic operations remains identical every time a model runs—we don't need the massive overhead of general-purpose computing (like branch predictors or speculative execution).

Hotz's thesis is that by stripping away this unnecessary complexity, we can create a "RISC" for ML. This isn't just about making things "smaller"; it's about making the relationship between the software and the hardware explicit. By simplifying the instruction set, we can generate the fastest possible programs for specific hardware, ensuring that the "petaflops" actually reach the people who need them rather than being lost in layers of abstraction.

Key ideas

The "RISC" of Machine Learning

• What it is — TinyGrad is a "Reduced Instruction Set" for ML models. It aims to be 10x less complex than current frameworks like PyTorch or XLA.
• Why it matters — By reducing the instruction set, you remove the "noise" of general-purpose computing, allowing for much tighter integration with hardware.
• In the book — Hotz compares the evolution of processors (from 360/x86 to RISC) to the current need for a simplified ML instruction set.

The Fallacy of "Completeness"

• What it is — Hotz argues that "completeness is harmful." In software, trying to make a framework do everything leads to a bloated, inefficient system that is harder to optimize.
• Why it matters — If a framework is too general, it becomes impossible to guarantee the most efficient execution path. You want a tool that does the specific job of ML perfectly, not a tool that tries to be a general-purpose programming language.
• In the book — He explains that CPUs devote massive silicon to branch predictors and reorder buffers because general code is unpredictable. Neural networks are not unpredictable; they are static loops.

Static Computation vs. Dynamic Branching

• What it is — Unlike general software, neural network loads and stores are static. The only thing that changes is the data, not the execution graph.
• Why it matters — Because the graph is static, we can optimize the entire execution path at compile time. We don't need the "safety nets" of dynamic execution because we know exactly what the model will do.
• In the book — Hotz points out that in a neural net, you load the same weight a million times; you don't need a branch predictor to tell you where the next weight is.

The Hardware-Software Co-dependency

• What it is — You cannot build a performant ML framework for a new chip if you haven't first mastered the framework for existing chips (like Nvidia's).
• Why it matters — Success in hardware requires a "full-stack" understanding. Google succeeded with TPUs because they wrote their own framework (TensorFlow) simultaneously.
• In the book — Hotz notes that if you can't write a performant framework for an AMD GPU, you have no hope of making a performant one for your own custom chip.

The "Tiny Box" Philosophy (Hardware Accessibility)

• What it is — The goal of "Tiny" is to create "luxury AI computers" for individuals—compact, powerful, and private.
• Why it matters — It democratizes access to high-end compute (like running Llama models locally) while ensuring privacy and independence from centralized cloud providers.
• In the book — He describes a vision of a "Tiny Box" that can run locally, potentially even participating in a private "mine flop" network where data integrity is enforced by hardware-bound private keys.

Frameworks & tools

The RISC-ML Design Principle

To build or evaluate an ML framework, use this checklist to determine if it is over-engineered:

1. Identify the Static Core: Does the framework acknowledge that neural network operations are static loops?
2. Remove Branching Overhead: Does the framework rely on dynamic branch predictors or speculative execution for standard ML ops? (If yes, it's over-engineered).
3. Minimize Instruction Count: Can the operation be expressed in fewer instructions than the current standard?
4. Explicit Memory Management: Does the framework allow for clear, visible memory access patterns (like debug=2 in TinyGrad)?

The "TinyGrad" Debugging Protocol

Use this to audit the efficiency of your model's execution:

1. Set graph.1: View the complete graph of all operations.
2. Set debug=2: Print the exact kernels dispatched to the GPU.
3. Analyze Flops/Bytes: Check the exact number of flops and memory accesses per kernel.
4. Identify Redundant Rounds: Look for unnecessary memory round-trips (e.g., reading/writing to memory when a kernel could have held the data).

Memorable examples & stories

The "One-Year Warranty" on Dev Kits

Hotz explains that "dev kits" are just consumer products with a different name. The gatekeeping isn't about the government; it's about whether the product is for you. Lesson: Don't let labels of "research" or "dev" distract from the reality of the product's utility.

The "Kernel Panic" with AMD

Hotz recounts his experience trying to get a pre-release version of ROCm to work, only to find it was riddled with kernel panics. He argues that "dumping source code on GitHub" isn't open source if the issues from 2022 are still there. Lesson: True open source requires active maintenance and a commitment to usability, not just a repository of abandoned code.

The "Humanoid" Robot on a Pole

Hotz describes his "Comma" robot: a camera on a pole with two wheels. It's the simplest possible physical manifestation to turn a robotics problem into a software problem. Lesson: Strip away the mechanical complexity to see if the software actually works.

Key quotes

"Completeness is harmful... it should be avoided." Why it matters: This challenges the standard software engineering goal of "feature-rich" libraries, arguing that for ML, more features often mean less efficiency.

"Neural networks are even better... no loads even depend on X... you do the exact same set of loads, stores, and arithmetic every single time." Why it matters: This is the fundamental technical justification for why we can optimize ML in ways we can't optimize general-purpose software.

"I want to make sure that [ML compute] can't happen structurally... I want to make sure that can't happen." (Regarding nationalization of compute). Why it matters: This explains the underlying "why" of TinyGrad—it's a move toward decentralized, accessible infrastructure.

How to apply it

This week

• Audit your current stack: Identify one part of your ML pipeline that feels "bloated" or unnecessarily complex.
• Run a "Debug 2" test: If using TinyGrad, run debug=2 on a simple model to see exactly what kernels are being dispatched.
• Identify Static Loops: Look at a piece of code you wrote and ask: "Is this actually dynamic, or is it a static loop where only the data changes?"

Ongoing

• Prioritize "Reduced" over "Complete": When choosing tools, choose the one that does the specific task with the fewest moving parts.
• Think in Information Theory: When designing systems, think about the flow of information and the constraints of the "pipe" (bandwidth, power, and memory).
• Software-First Robotics: When approaching physical problems, always ask: "How can I turn this into a software problem by simplifying the hardware?"

Common pitfalls

• The "Convenience" Trap: It is easy to use PyTorch because it is "complete," but that completeness comes at the cost of hidden inefficiencies.
• The "GPU-only" Fallacy: Assuming that because a chip is good at training, it is automatically good at everything. Hotz notes that Google's success was tied to their ability to write a custom framework for their specific hardware.
• Mistaking "Open Source" for "Public Source": Don't assume a project is open source just because the code is on GitHub; if it's broken and unmaintained, it's just a public graveyard.

Connects to

• RISC-V Architecture: The underlying philosophy of "Reduced Instruction Sets" is what Hotz is applying to the ML software stack.
• Information Theory: Hotz views his work through the lens of information theory (how data moves and is processed) rather than just pure physics.

If you remember only three things

• ML is Static: Unlike general software, the execution graph of a neural network is fixed; we can exploit this to remove unnecessary "dynamic" overhead.
• RISC for ML: We need a simplified instruction set for machine learning to achieve true hardware efficiency.
• Completeness is a Liability: A framework that tries to do everything is harder to optimize; focus on the "Reduced" set of operations that actually matter.

Internalize it

• Reflection questions
  • If I were to strip away 90% of the features in my current software stack, what would be the 10% that actually perform the core task?
  • Why am I using a "complete" library for a task that is fundamentally a "static" loop?
  • How does the "RISC" philosophy of hardware design apply to the way I organize my own mental models or workflows?
• Flashcards Q: What is the primary difference between general CPU execution and Neural Network execution? A: General CPU execution requires branch predictors and speculative execution because code is dynamic; Neural Networks are static loops where the sequence of operations is fixed. Q: Why does Hotz argue that "completeness" is harmful in ML frameworks? A: Because a "complete" framework is often over-engineered with features that add complexity and overhead to a task that doesn't require them. Q: What is the "RISC" equivalent for ML? A: A Reduced Instruction Set for ML (like TinyGrad) that focuses on a minimal, high-performance set of operations. Q: What was the key to Google's success with TPUs? A: They didn't just build a chip; they built a custom ML framework (TensorFlow) specifically for that chip. Q: What does "static" mean in the context of neural network loads? A: It means the same set of loads, stores, and arithmetic operations are performed every single time the net is run; only the data changes.

Fidelity note

Confidence: High. This summary is grounded strictly in the provided transcript. The distinction between "CISC" and "RISC" as applied to ML is a core theme of the conversation. The "Tiny Box" and "RISC-ML" concepts are directly attributed to Hotz's vision for TinyGrad. One nuance: Hotz mentions "mining flop coin" as a potential use case for the "Tiny Box," which is a speculative future application of his hardware.