← Originals
Ilya Sutskever

Ilya Sutskever

6 interviews

What they argue

Ilya Sutskever is a leading AI researcher and thinker who operates at the intersection of architectural innovation, scaling laws, and the fundamental principles of machine learning. He focuses on the transition from pure scaling to a research-driven era where the goal is to discover how large neural networks develop complex reasoning and judgment.

  1. The Transition to the Research Era

    The 'Age of Scaling' is reaching its limits because more data and compute do not inherently guarantee the emergence of taste or judgment. The next era of AI development will focus on discovering the fundamental principles of learning and alignment.

    From Ilya Sutskever – We're moving from the age of scaling to the age of research (episode)

    • A related idea Dario Amodei — The Scaling HypothesisThe rapid advancement of AI is driven by the principle that increasing model size, data, and compute in a synchronized, linear fashion leads to emergent intelligence.…#452 – Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity
    • A related idea Andrej Karpathy — Scaling and ShapingAI progress is driven by scaling up supervised learning (increasing parameters and data) and 'Shaping,' which is the professional practice of refining agent behaviors…Short Story on AI: A Cognitive Discontinuity.

  2. The Philosophy of Scale

    The current AI revolution is rooted in a shift from 'proving' to 'building' rather than focusing on mathematical proofs. Sufficiently large neural networks trained on enough data will inevitably develop unprecedented capabilities by solving complex objectives.

    From No Priors Ep. 39 | With OpenAI Co-Founder & Chief Scientist Ilya Sutskever (episode)

    • Says much the same Dario Amodei — The Scaling HypothesisThe rapid advancement of AI is driven by the principle that increasing model size, data, and compute in a synchronized, linear fashion leads to emergent intelligence.…#452 – Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity
    • A related idea Andrej Karpathy — Scaling and ShapingAI progress is driven by scaling up supervised learning (increasing parameters and data) and 'Shaping,' which is the professional practice of refining agent behaviors…Short Story on AI: A Cognitive Discontinuity.

  3. The 'Small Little Brains' Philosophy

    Neural networks should be treated as biological-adjacent systems rather than purely mathematical abstractions. This shifts the research goal toward understanding how these systems behave as they scale.

    From No Priors Ep. 39 | With OpenAI Co-Founder & Chief Scientist Ilya Sutskever (episode)

    • A related idea Daniel J. Siegel — The Social BrainThe human brain is uniquely designed for social connection, and our evolution has favored those who could navigate complex social hierarchies. This means that play and…The developing mind

  4. Architectural Efficiency and Data-Driven Priors

    The primary levers for scaling are architectural efficiency and data-driven priors rather than just increasing size. Research is moving toward understanding how 'compression' and 'registers' enable complex reasoning.

    From ICLR 2024 — Best Papers & Talks (ImageGen, Vision, Transformers, State Space Models) ft. Durk Kingma, Christian Szegedy, Ilya Sutskever (episode)

    • A related idea Dylan Patel — Physical Infrastructure ConstraintsScaling AI compute is an industrial engineering problem where the primary bottlenecks are physical resources like clean room capacity, tooling, and power generation.…Dylan Patel — The single biggest bottleneck to scaling AI compute

  5. The Register Mechanism in Vision Transformers

    Registers are additional learnable tokens appended to the input sequence of a Vision Transformer during training. They provide a dedicated space to handle high-norm tokens in low-informative background areas of feature maps.

    From ICLR 2024 — Best Papers & Talks (ImageGen, Vision, Transformers, State Space Models) ft. Durk Kingma, Christian Szegedy, Ilya Sutskever (episode)

  6. The Disconnect Between Eval Performance and Real-World Utility

    Models can become 'superhuman' at specific tasks, such as coding competitions, without gaining the underlying judgment required to be useful. True progress should be measured by whether a model performs better on a new task after learning.

    From Ilya Sutskever – We're moving from the age of scaling to the age of research (episode)

    • Says much the same Melanie Mitchell — Performance vs. CompetenceCurrent AI systems often excel at mimicking the outputs of intelligent behavior without possessing an internal model of reality. We must distinguish between a system's…Artificial Intelligence: A Guide for Thinking Humans
    • A related idea Nathan Labenz — The Shift from Prediction to Task CompletionModern AI models are moving beyond simple next-token prediction toward being trained to achieve specific goals via reinforcement learning (RL). This shift means models…AI Scouting Report: the Good, Bad, & Weird @ the Law & AI Certificate Program, by LexLab, UC Law SF

  7. The Scaling Logic Framework

    To determine if a project is worth pursuing, it should be evaluated on its scalability property and its ability to improve as more compute and data are added. It also considers whether the model can perform a task that is impossible for a human to do.

    From No Priors Ep. 39 | With OpenAI Co-Founder & Chief Scientist Ilya Sutskever (episode)

Every idea names the summary it came from. Written by a local model from this mind’s own work, never from memory.

Interviews 3

Open Ilya Sutskever in Originals