What Is ChatGPT Doing — and Why Does It Work

In one sentence

The success of ChatGPT proves that human language is fundamentally simpler than it appears, allowing large neural networks to learn "semantic laws of motion" within a high-dimensional feature space.

Why read this (and who it's for)

This book strips away the marketing hype to reveal the actual mechanics of Large Language Models (LLMs). It solves the mystery of why an AI can write a coherent essay but often fails at simple logic. You will benefit most if you want to understand the architectural limits of AI, how "meaning" is mathematically represented, and why certain tasks remain "computationally irreducible."

The big idea

ChatGPT does not "think" in the human sense; it produces a "reasonably continuation" of text by predicting the next token based on probabilities. Because humans have written billions of pages of text, there are underlying regularities—or "semantic laws"—that govern how meaning moves from one word to the next.

The central thesis is that language is "computationally shallower" than we assumed. Because it follows these regularities, a neural network with enough "knobs" (parameters) can learn to navigate this space. However, there is a fundamental limit: because some processes are "computationally irreducible," the model cannot guarantee an outcome without doing the actual work; it can only provide what "sounds right" based on the statistics of conventional wisdom.

Key ideas

Reasonable Continuation

The core mechanism of ChatGPT is to produce a "reasonably continuation" of existing text. It doesn't look at literal text but for things that "match in meaning," producing a ranked list of probabilities for the next token. Why it matters — This means the AI isn't retrieving facts; it is navigating a probability map. If you understand this, you realize why it can hallucinate—it is simply following the most likely trajectory of words, not the most "true" one. In the book — The author explains that picking only the highest-ranked word creates "flat," uncreative text; instead, a "Temperature Parameter" allows for lower-ranked words to be picked at random to produce "more interesting" essays.

The $n$-gram Problem & LLM Solution

Predicting text by counting occurrences of sequences (2-grams, 3-grams) is impossible because the number of possibilities exceeds the particles in the universe. Why it matters — This explains why we need "Large" Language Models. They are built to estimate probabilities for sequences that have never been explicitly seen in a training corpus. In the book — The author notes that with ~40,000 common words, 3-grams alone result in 60 trillion possibilities, making simple counting impossible.

Meaning Space (Embeddings)

An embedding is an array of numbers representing the "essence" of something where "nearby things" are represented by nearby numbers. This creates a "Meaning Space." Why it matters — It allows the AI to understand synonyms and nuances without explicit rules. If two words appear in similar environments, they are placed nearby in this space. In the book — The author uses the example of "alligator" and "crocodile" appearing in similar environments $\rightarrow$ placed nearby; "turnip" and "eagle" do not $\rightarrow$ placed far apart.

Computational Irreducibility

A state where there is no shortcut to determine a result; one must explicitly perform the computation to see what happens. Why it matters — This defines the limit of AI capability. The more you want a system to make "true use" of its capabilities, the more it shows irreducibility, and the less it's going to be trainable. In the book — The author argues that we can never guarantee that the "unexpected won't happen" without doing the actual work; human brains evolved to avoid these specific types of computations.

Syntactic vs. Semantic Grammar

Syntactic grammar refers to rules for construction (nouns, verbs), while semantic grammar requires a "model of the world" to serve as a "skeleton" for layered language. Why it matters — This explains why ChatGPT can write an essay (syntactic) but struggle with complex formal logic or physical reality (semantic). In the book — The author notes that a sentence can pass semantic grammar but not be realized in the actual world (e.g., "The elephant traveled to the Moon").

Frameworks & tools

The Neural Net Training Process

To train a network like ChatGPT, follow these steps:

1. Supply "input $\rightarrow$ output" examples (e.g., unmasked text).
2. Compute a "loss function" (the sum of squares of differences between actual and true values).
3. Use the "chain rule" of calculus to "unravel" operations across layers.
4. Progressively adjust weights via "steepest descent" to minimize the loss.

The Human Feedback Loop (RLHF)

To move beyond "raw training":

1. Let the neural net finish its initial training on a huge corpus.
2. Have humans interact with the model and provide ratings.
3. Train a second neural net to predict those human ratings.
4. Use this prediction model as a loss function to fine-tune the original network.

Memorable examples & stories

The Voronoi Diagram Analogy: Recognition works by ensuring inputs are "attracted to one place" (e.g., all 1s go to one spot). It is like a Voronoi diagram where a point is assigned to the closest coffee shop. Lesson: Neural nets work by creating "attractor basins" for concepts.

Rule 30 & Complexity Amplification: Computational processes can greatly amplify apparent complexity even when underlying rules are simple (like Rule 30). Lesson: Language may be simple at its core, but it looks complex because of how those simple rules interact.

The Parenthesis Failure: When training a net on ( and ) sequences, it converges quickly with 2 attention blocks. However, it is "too computationally shallow" to explicitly count many parentheses in long sequences. Lesson: Even if a model learns a rule, it may lack the computational depth to apply it to complex, irreducible scales.

The "Alligator" Embedding: Because "alligator" and "crocodile" appear in similar environments, they are projected close together in feature space. Lesson: Meaning is derived from environmental context, not just dictionary definitions.

Key quotes

"It's Just Adding One Word at a Time" Why it matters: Strips away the magic; it’s a simple iterative probability calculation.

"There’s never a ‘model-less model’" Why it matters: Every AI system has an underlying structure and "knobs you can turn."

"Writing an essay turns out to be a ‘computationally shallower’ problem than we thought." Why it matters: Explains why LLMs are so good at creative writing despite lacking "reasoning."

"Computational irreducibility means that we can never guarantee that the unexpected won't happen." Why it matters: Highlights the fundamental safety and reliability limits of neural networks.

How to apply it

This week

• Identify a task you want an AI to do and determine if it is "shallow" (summarizing, stylistic rewriting) or "deep/irreducible" (complex formal logic, multi-step planning).
• When using an LLM for a complex task, provide external computational tools (like WolframAlpha or a calculator) because the model lacks "computational superpowers."

Ongoing

• View ChatGPT as a "trajectory" generator: It follows a path in "Meaning Space" based on your prompt.
• Recognize that the AI's "knowledge" is a distributed encoding of human text, not an internal model of reality.

Common pitfalls

• Thinking it has a "Model of the World": It actually only has a "skeleton" of semantic grammar; it doesn't know what is possible in the real world.
• Expecting perfect logic: Because of computational irreducibility, it cannot guarantee correct results for complex deductions without explicit recomputation.
• Assuming it "understands": It is simply navigating a feature space where "meaning" is represented by coordinates.

Connects to

• Rule 30: Explains how simple rules can create complexity amplification.
• Wolfram Language: Represents the "computational language" required for unambiguous execution on a computer, contrasted with human social contracts of language.

If you remember only three things

• ChatGPT works by predicting the next token ("reasonably continuation") based on probabilities in a high-dimensional "Meaning Space."
• Human language is fundamentally simpler than it seems, allowing neural networks to learn its "semantic laws of motion."
• Because some processes are "computationally irreducible," LLMs can only produce what "sounds right" and cannot guarantee correctness for complex logic.

Internalize it

• Reflection questions

  • Which tasks in my daily work are "shallow" (easy for an LLM) and which are "irreducible" (requiring explicit computation)?
  • If I treat ChatGPT as a trajectory generator, how does that change the way I write prompts?
  • How can I provide "external computational tools" to help the AI overcome its lack of "computational superpowers"?

• Flashcards Q: What is the fundamental goal of ChatGPT? A: To produce a "reasonably continuation" of existing text based on probabilities. Q: What is an "embedding"? A: An array of numbers representing the "essence" of something where nearby meanings are represented by nearby numbers. Q: What does "computational irreducibility" mean? A: A state where there is no shortcut to a result; one must explicitly perform the computation to see what happens. Q: Why did ChatGPT succeed at writing essays? A: Because writing an essay is a "computationally shallower" problem than previously assumed. Q: What is the difference between Syntactic and Semantic Grammar? A: Syntactic grammar is construction rules; Semantic grammar requires a "model of the world" as a skeleton.

Fidelity note

Confidence: High. This summary is grounded strictly in the provided notes from sections 1–6. I am certain about the definitions of Meaning Space, Computational Irreducibility, and the mechanism of "Reasonable Continuation." The distinction between Syntactic and Semantic grammar is also clearly defined in the text.