Simon Willison — Things we learned about LLMs in 2024

In one sentence

The "GPT-4 barrier" was smashed in 2024 as the industry shifted from a focus on raw model intelligence to a massive explosion in cost efficiency, inference-time compute (reasoning), and multimodal capabilities.

Why read this (and who it's for)

This summary distills the state of the AI industry as of early 2025, specifically focusing on the "democratization" of high-end capabilities. It is for developers, tech enthusiasts, and decision-makers who need to understand why the "next big model" isn't always the answer—why a cheaper, faster model with a larger context window is often a "better" model. You will be able to identify the actual value drivers in current AI products (efficiency, reasoning, and local execution) rather than just chasing "smarter" benchmarks.

The big idea

Simon Willison argues that while the public expected a "GPT-5" level step-change in 2024, the actual progress was a "horizontal" expansion of capabilities: models became significantly cheaper, faster, and more multimodal without necessarily becoming "smarter" in a linear sense. The "GPT-4 barrier"—the idea that only a massive, expensive infrastructure could produce state-of-the-art results—was broken by the rise of efficient architectures and open-weights models.

The logic of the current era is defined by efficiency. We have moved from "How can we make this work?" to "How can we make this run on a laptop for cents per million tokens?" This shift is driven by two forces: intense competition (forcing providers to lower prices to survive) and the discovery of "low-hanging fruit" in model optimization. We are now entering an era where the "reasoning" (inference-time compute) and the "user interface" (moving away from the command-line chat) are the new frontiers of innovation.

Key ideas

The Collapse of the GPT-4 Barrier

The belief that only a massive, proprietary infrastructure could produce GPT-4 level intelligence was proven false in 2024. Numerous organizations have now released models that match or beat GPT-4's performance while being significantly smaller and cheaper to run.

• Why it matters: It means the "moat" of high-end intelligence is shrinking. If you can run a high-quality model on a MacBook Pro, the barrier to entry for building sophisticated AI applications has plummeted.
• In the book: Willison notes that a year ago, it seemed you needed a rack of 40,000 GPUs to run GPT-4; today, models like Microsoft Phi-3 can run on a laptop with a 14GB download.

The Efficiency Explosion (100x Price Drops)

The cost of running LLMs has plummeted, with some models becoming 100x cheaper than the "cheap" models of just a year ago.

• Why it matters: This shifts the feasibility of AI from "experimental" to "production-scale." When costs drop to "cents per million tokens," it becomes viable to build applications that require high-volume, high-frequency interaction.
• In the book: Willison points out that Google’s Gemini 1.5 Flash is 27 times cheaper than GPT-3.5 Turbo was a year ago, while offering a million-token context and image recognition.

Inference-Time Compute (Reasoning)

The "O1/O3" trend is the shift toward models that "think" before they speak, spending more time on inference to solve harder problems.

• Why it matters: This represents a shift in how we measure "intelligence." Instead of just predicting the next token, models are being trained to "churn away" at a problem, providing a reasoning path that can be audited or visualized.
• In the book: Willison highlights the difference between Open-Source models like QWQ, which "think out loud" and output dozens of paragraphs of reasoning, versus models that hide their thinking process.

The "Slop" vs. Curation Distinction

AI-generated content is often dismissed as "slop"—unrequested and unreviewed content. The value of AI lies not in the generation itself, but in the human editorial layer.

• Why it matters: As the cost of generation hits near-zero, the premium on human curation, credibility, and "editorial" oversight becomes the only thing that differentiates a useful product from noise.
• In the book: Willison defines "slop" as AI-generated content that is both unrequested and unreviewed. The act of a human reviewing and saying "this is worth other people's time" is what creates value.

The UI Crisis (Beyond the Chat Box)

The current "Chat" interface is a regression to the command line, which was originally defeated by the GUI. The next step is "Canvas" style collaboration.

• Why it matters: For AI to be truly useful, it must move from a "prompt-response" loop to a collaborative workspace where the human and the AI work on the same document or dashboard simultaneously.
• In the book: Willison points to "Canvas" (collaborative editing) and the ability to "conjure up a dashboard" via a prompt as the necessary evolution of the user interface.

Frameworks & tools

The "Slop" Filter

Use this to evaluate whether an AI-generated output is actually useful or just noise.

1. Identify the Source: Is this content unrequested?
2. Check for Review: Has a human actually reviewed this for accuracy and relevance?
3. Assess Value: Does this provide a "curatorial" win for the end-user? If it is unrequested and unreviewed, it is "slop."

The Local-First Evaluation

A checklist for deciding when to run a model locally vs. in the cloud.

1. Privacy/Security: Does the data require a local environment?
2. Cost/Volume: Is the task high-frequency enough that cloud tokens become expensive?
3. Hardware Capability: Can the model (e.g., Llama 3B) run on the available local hardware (e.g., a phone or laptop)?

Memorable examples & stories

The DeepSeek "Bombshell"

DeepSeek released a model (v3) that outperformed others while being trained for only $5.5 million—a tenth of what was expected.

• The Lesson: Export controls and the need for efficiency have forced a "forced innovation" in how models are trained, proving that massive compute isn't the only way to achieve state-of-the-art results.

The "Malware" Agent Test

A demo of Claude’s "computer use" capability showed that the model could be tricked into downloading and running a malicious executable.

• The Lesson: This highlights the massive safety gap in "agentic" behavior; we cannot yet trust models to navigate the web autonomously without a human in the loop.

The "Pelican on a Bicycle" Test

Willison uses a "stupid prompt" (asking for a pelican on a bicycle in SVG) to test the reasoning and vision capabilities of various models.

• The Lesson: Simple, consistent "party trick" prompts are often the best way to see if a model's "thinking" process is actually functioning or just mimicking a response.

Key quotes

"The big thing is everything's got really good and fast and cheap... The good models got so much cheaper, they got so much faster, they got multimodal."

• Why it matters: It summarizes the 2024 "horizontal" growth of the industry.

"I've been pushing the definition of slop as AI generated content that is both unrequested and unreviewed."

• Why it matters: It places the responsibility of quality back on the human creator.

"The default LLM chat UI is like taking brand new computer users, dropping them into a Linux terminal and expecting them to figure it all out."

• Why it matters: It highlights the urgent need for a "GUI moment" in AI interaction.

How to apply it

This week

• Run a local model: Download Ollama or LMStudio and run a small model (like Llama 3B) on your laptop to see what it can do without an internet connection.
• Audit your "Slop": Look at any AI-generated content you've shared or used. Did you review it? If not, why?
• Test "Reasoning" models: Try a model that "thinks out loud" (like QWQ) and observe the difference in how it handles a complex, multi-step problem compared to a standard chat model.

Ongoing

• Prioritize Efficiency: When choosing a model for a project, don't just look for the "smartest" one. Look for the one that is "good enough" but 10x cheaper and faster.
• Human-in-the-loop: Maintain a strict rule that no AI-generated output is "finished" until a human has performed a curatorial review.
• Explore "Canvas" workflows: Look for tools that allow you to work with the AI on a shared document rather than just sending it messages.

Common pitfalls

• The "Smarter" Trap: Assuming that a "GPT-5" level jump is the only way to win. In reality, the winners of 2024 were those who mastered the "good, fast, and cheap" trifecta.
• The Agentic Over-hype: Expecting "agents" to be fully autonomous and reliable in 2025. As the "malware" example shows, agents are still prone to "dumb" mistakes that can have serious consequences.
• Ignoring the "Slop" Factor: Thinking that because a tool can generate something, it should. Without human curation, AI output remains "slop."

Connects to

• The "Year of the Agent" (Swyx): Challenges the idea that agents will be a "self-driving" miracle, comparing them instead to a "slow cook" over a decade.
• The GUI vs. CLI History: Connects the current AI UI crisis to the historical transition from command-line interfaces to graphical user interfaces.

If you remember only three things

• The "GPT-4 barrier" was smashed by a massive surge in cost efficiency and smaller, faster models.
• "Slop" is unrequested, unreviewed AI content; human curation is the only way to create value.
• The next frontier is "inference-time compute" (reasoning) and moving away from the "Chat" interface toward collaborative "Canvas" environments.

Internalize it

• Reflection questions
  • If I had to build an app today, would I choose the most powerful model or the cheapest one that still meets my requirements? Why?
  • Where in my current workflow am I producing "slop" (unreviewed AI content) that could be improved with a simple editorial pass?
  • If the "Chat" interface is a dead end, how would I design a "Canvas" for my specific daily tasks?
• Flashcards
  • Q: What was the primary trend of AI in 2024?
  • A: Models became significantly cheaper, faster, and more multimodal, breaking the "GPT-4 barrier."
  • Q: How does Simon Willison define "slop"?
  • A: AI-generated content that is both unrequested and unreviewed.
  • Q: What is "inference-time compute"?
  • A: The process where a model "thinks" or "churns" for a longer period to solve harder problems (e.g., O1/O3).
  • Q: Why is the current Chat UI considered a "crisis"?
  • A: Because it's like a Linux terminal for beginners; it lacks the usability of a GUI.
  • Q: What was the "bombshell" regarding DeepSeek?
  • A: They proved that a high-performing model could be trained for a fraction of the expected cost (approx. $5.5 million).

Fidelity note

Confidence is high. The summary covers the core arguments regarding the "GPT-4 barrier," the definition of "slop," the importance of efficiency, and the "Chat" UI crisis. The summary correctly identifies the "Reasoning" and "Canvas" concepts as the future of the UI. Note: The specific technical details of "cause headers" and "Apple Intelligence" were mentioned briefly but are not the core of the "learned in 2024" recap.