← Originals
Andrej Karpathy

Andrej Karpathy

15 essays · 6 interviews

ai · business

What they argue

Andrej Karpathy is a technical thinker who operates at the intersection of machine learning, software engineering, and the underlying mathematical principles of artificial intelligence. He focuses on deconstructing the 'black box' of neural networks to reveal the fundamental mechanics, data-centric architectures, and computational theories that drive modern AI.

  1. Leaky Abstractions in Neural Networks

    Unlike standard software where APIs hide complexity, neural network components like Batch Norm or RNNs do not magically solve problems and can fail silently. Success requires being thorough and obsessed with visualizations of every possible component.

    From A Recipe for Training Neural Networks (essay)

    • A related idea George Hotz — The RISC-ML Design PrinciplesBecause neural networks are fundamentally static and predictable, we can achieve superior performance by stripping away the 'Turing-complete' overhead of current…Commoditizing the Petaflop — with George Hotz of the tiny corp

  2. Software 2.0

    This is a new programming paradigm where the programmer specifies a goal rather than writing explicit instructions. The computer then searches for optimal weights within a neural network architecture to achieve that goal.

    From Software 2.0 (essay)

  3. Macro Stability of Deep Learning

    The fundamental architecture of neural networks—layers, neurons, and backpropagation—has remained remarkably stable for over three decades. Most 'new' architectures are actually iterative improvements on this core structure.

    From Deep Neural Nets: 33 Years Ago and 33 Years From Now (essay)

  4. Optimization over Programs

    Recurrent Neural Networks (RNNs) differ from standard networks by maintaining an internal state that evolves over time. This allows them to treat training as an optimization over programs rather than just mapping fixed-sized inputs to outputs.

    From The Unreasonable Effectiveness of Recoded Neural Networks (essay)

    • Pushes the other way George Hotz — The RISC-ML Design PrinciplesBecause neural networks are fundamentally static and predictable, we can achieve superior performance by stripping away the 'Turing-complete' overhead of current…Commoditizing the Petaflop — with George Hotz of the tiny corp

  5. Scaling and Shaping

    AI progress is driven by scaling up supervised learning (increasing parameters and data) and 'Shaping,' which is the professional practice of refining agent behaviors using specific curriculums. This creates agents that excel at reflexes but may still face a 'cognitive discontinuity' in complex reasoning.

    From Short Story on AI: A Cognitive Discontinuity. (essay)

    • A related idea Ilya Sutskever — The Philosophy of ScaleThe current AI revolution is rooted in a shift from 'proving' to 'building' rather than focusing on mathematical proofs. Sufficiently large neural networks trained on…No Priors Ep. 39 | With OpenAI Co-Founder & Chief Scientist Ilya Sutskever
    • A related idea Dario Amodei — The Scaling HypothesisThe rapid advancement of AI is driven by the principle that increasing model size, data, and compute in a synchronized, linear fashion leads to emergent intelligence.…#452 – Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity

  6. The Energy Hierarchy

    Human metabolism can be understood as a system of energy conversion where fat is burned as combustion and ATP acts as a universal energy currency. This can be mapped to a computer analogy: Phosphocreatine as L1/L2 Cache, Glycogen as RAM, and Adipose Tissue as the hard drive.

    From Biohacking Lite (essay)

    • A related idea Josh Axe — Mitochondrial Energy ProductionEnergy is not a finite resource you are born with, but is manufactured moment-by-moment inside your cells by mitochondria. Fatigue is a physiological drain caused by…The Real Reason You Wake Up Tired Every Morning

  7. Blockchain as Open Source + State

    Blockchain is the extension of software development to include a shared, persistent state that is accessible to everyone. It shifts the paradigm from merely sharing code to sharing a 'running computer.'

    From A from-scratch tour of Bitcoin in Python (essay)

Every idea names the summary it came from. Written by a local model from this mind’s own work, never from memory.

Essays 13

Interviews 3

Open Andrej Karpathy in Originals