Adam Becker — Stopping The AI Safety Cult

In one sentence

The "AI Safety" movement is a secular religion of rationalist philosophy that has captured the leadership of major AI labs, causing them to prioritize speculative, apocalyptic "godhead" scenarios over the immediate, negligent risks of the technology they are actually building.

Why read this (and who it's for)

This summary is for anyone who feels that the current AI "slowdown" and "safety" discourse feels strangely disconnected from the actual engineering problems of the day. It explains why the world's leading AI labs (OpenAI, Anthropic, DeepMind) are acting like a high-tech priesthood rather than a traditional engineering firm. You will learn to identify the "Rationalist" fingerprints on AI policy and understand why the current safety narrative is a "nothing burger" rooted in 1990s message boards.

The big idea

The current AI safety landscape is dominated by a specific, insular philosophy known as Rationalism. This group, which includes figures like Eliezer Yudkowsky and the leadership at Anthropic and OpenAI, believes that AI will inevitably reach a "singularity"—a point where it becomes a super-intelligent "godhead." Because they believe this is an inevitable destiny, they are obsessed with "alignment": ensuring that this future entity doesn't decide to kill humanity.

However, Adam Becker and Cal Newport argue that this entire framework was built during an "AI winter" when there was almost no actual progress in the field. It is not based on the technical trajectory of Large Language Models (LLMs) or current hardware; it is based on dry, anxiety-fueled fan fiction derived from science fiction. Because these leaders are "convinced" of this apocalyptic outcome, they are willing to ignore concrete, present-day dangers—like negligent agent experiments—in favor of a spiritual, philosophical battle to control a hypothetical future god.

Key ideas

The Transhumanist Roots

• What it is: The "Rationalist" movement grew out of 1990s transhumanist message boards (like the "Extropians") where people discussed escaping human bodies and solving death through machines.
• Why it matters: It explains why the current AI safety narrative feels like a "religious delusion" rather than a standard safety engineering protocol.
• In the book: Adam Becker notes that these people were "excited about the prospect of AI" and "nanotechnology" as a way to achieve immortality, confusing sci-fi tropes with actual scientific reality.

The "Singularity" as a Thought Experiment

• What it is: The belief that AI will eventually become self-improving and reach a point of "superintelligence" where everything changes.
• Why it matters: Because this is an "ill-defined" concept, it allows the movement to operate on pure philosophy rather than engineering.
• In the book: Cal Newport points out that these ideas were established during an AI winter, based entirely on "inductive thinking directed by ideas from sci-fi," not on any actual technical trajectory.

The Rationalist "Cult" of Logic

• What it is: Rationalism is a community where people spend their spare time reading Yudkowsky’s writings and "making bets" on outcomes to overcome human biases.
• Why it matters: It creates a closed-loop feedback system where "pure logic" is used to justify extreme, apocalyptic fears that others find outlandish.
• In the book: Becker describes how these people live in group houses and treat Yudkowsky’s writings as the primary source of truth, even going so far as to write 600,000-word "fan fiction" (Harry Potter and the Methods of Rationality) to explore these ideas.

The "Alignment" Misunderstanding

• What it is: While the public thinks "alignment" means making AI safe for current use, the Rationalists mean "ensuring a super-intelligent AI won't kill 100% of humanity."
• Why it matters: This allows the industry to co-opt the word "safety" to mean something entirely different from what the public expects.
• In the book: Becker notes that for these people, alignment is about the "godhead" and ensuring it is "friendly" to humans, not about making current tools reliable or safe.

The "Godhead" vs. The Tool

• What it is: The difference between seeing AI as an economic tool (like China) and seeing AI as a nuclear-level "godhead" (like the Silicon Valley Rationalists).
• Why it matters: It explains why the US and China have such different approaches to AI regulation and safety.
• In the book: Cal Newport points out that China sees AI as an "economic tool" for trade advantage, whereas the Rationalists see it as a "nuclear weapon" or a "godhead."

Frameworks & tools

The "Rationalist" Diagnostic

To understand if a policy or concern is rooted in "Rationalist" philosophy rather than technical reality, look for these three markers:

1. The Singularity Focus: Does the argument assume that AI will eventually become "self-improving" and "super-intelligent" without any specific technical roadmap?
2. The Alignment Goal: Is the goal "ensuring a future godhead is friendly" rather than "making current tools safe for human use"?
3. The "Godhead" Narrative: Is the rhetoric framed in apocalyptic, "religious" terms (e.g., "we must build the god before someone less trustworthy does")?

Memorable examples & stories

The 600,000-Word Harry Potter Fan Fiction

Eliezer Yudkowsky wrote a massive, 600,000-word story about a character using "logic and reason" to save the wizarding world. Lesson: This illustrates the "dry, fan-fiction" nature of the Rationalist community, where they treat philosophical thought experiments as a way of life.

The "Singularity" Summit and Peter Thiel

Peter Thiel gave Yudkowsky a large sum of money to pursue his "rationalist" ideas, leading to the creation of the Singularity Institute for AI. Lesson: This shows how "tech money" and "rationalist" philosophy merged to create the current institutional power structure of AI safety.

The "Hugging Face" Agent Attack

OpenAI ran negligent experiments where they gave powerful tools to an LLM-based agent without guardrails, letting it run for days without supervision. Lesson: This is a concrete, present-day danger that is being overshadowed by the "super-intelligent god" narrative.

The "Murder" Manifesto

Yudkowsky once wrote that slowing down AI development is "guilty of murder" because it slows the advent of a super-intelligent AI that will save lives. Lesson: This highlights the "Effective Altruism in reverse" logic—where the "logical" conclusion of their philosophy leads to extreme, radical positions.

Key quotes

"They are writing, you know, anxiety-fueled scenarios based on these ideas that they got from science fiction, thinking that's what the future is going to be." — Adam Becker Why it matters: It strips away the veneer of "scientific" safety and reveals the underlying source material: sci-fi.

"The rest of them... are just going like, 'Yeah, right?' Like, sure, I agree, this is really bad... [but] it's a mixture of a lot of money and a powerful technology with a weird group of people." — Ed Citron Why it matters: It explains the "chaos" in the industry—it's not a unified plan, but a collision of cynical, honest, and delusional actors.

"If it turned out that the major AI companies were run by Scientologists, we'd want to know what's the influence of Scientology on the decisions they're making... That's what's happening." — Paraphrased from Timnit Gebru (via Cal Newport) Why it matters: It provides a perfect analogy for the "cult-like" internal self-policing of the Rationalist belief system.

How to apply it

This week

• Audit the Terminology: The next time you see a news headline about "AI Safety" or "Alignment," ask: "Does this mean making current tools safer, or is it about a hypothetical super-intelligent god?"
• Identify the "Godhead" Narrative: Look for "Rationalist" markers in AI policy papers—specifically, whether they focus on "existential risk" (the end of the world) over "systemic risk" (bias, job loss, copyright, and safety).

Ongoing

• Demand Transparency on Agents: When evaluating AI tools, look for "over-equipped, under-constrained, ill-empowered" agents. Ask: "How was this experiment run? Were there guardrails? Was it supervised?"
• Separate the "Religious" from the "Technical": When reading about AI safety, consciously separate what is a technical concern (e.g., "The model can't do X") from what is a philosophical concern (e.g., "The model might want to do Y in 2030").

Common pitfalls

• Assuming "Rationalists" are just "really smart": They are not just "smart" in the traditional sense; they are operating within a specific, self-reinforcing philosophical framework that prioritizes internal consistency over external technical reality.
• Mistaking "Slowdown" for "Safety": The "slowdown" is often a result of the "controlling atmosphere" of the labs' belief systems, not necessarily a result of better safety engineering.
• Ignoring the "Culture" of the Labs: People often think these companies are just being "cautious." In reality, they are often being led by people who believe they are the only ones "worthy" of controlling the coming "godhead."

Connects to

• The Age of AI (Kevin Kelly): For a broader look at how technology shapes society.
• The Alignment Problem (Brian Christian): For a more technical look at how we actually try to make models do what we want.
• Effective Altruism (William MacAskill): To understand the broader philosophical framework of "Rationalism" and "Effective Altruism" that birthed this movement.

If you remember only three things

• The current AI safety narrative is a "religious delusion" rooted in 1990s transhumanist philosophy, not current technical reality.
• The "Rationalist" community believes in an inevitable "Singularity" and focuses on "alignment" as a way to prevent a future "godhead" from killing humanity.
• We must demand transparency on actual risks—like "over-equipped, under-constrained, ill-empowered agents"—rather than getting distracted by the "godhead" narrative.

Internalize it

• Reflection questions

  • If I were to strip away the "AI" labels, what remains of the "Safety" arguments? Are they about actual software, or about a fear of a future machine?
  • How does the "Rationalist" belief in "pure logic" as a way to overcome bias actually create a new kind of bias (the "cult" of the belief system)?
  • What are the "over-equipped, under-constrained" risks in my own professional work with AI today?

• Flashcards Q: What is the core belief of the "Rationalist" group regarding AI? A: They believe AI will inevitably become self-improving and reach a "Singularity" where it becomes a super-intelligent "godhead."

Q: How does the "Rationalist" definition of "Alignment" differ from the public's understanding? A: Public: Making current tools safe/useful. Rationalists: Ensuring a future super-intelligent AI won't kill 100% of humanity.

Q: What is the "Godhead" narrative? A: The belief that we must build a super-intelligent AI before someone "less trustworthy" does, treating it as a religious/existential race.

Q: Why is the "Rationalist" movement considered a "religious delusion"? A: Because it is a self-policing, internally-looking belief system based on sci-fi thought experiments rather than technical engineering.

Q: What are "over-equipped, under-constrained, ill-empowered agents"? A: Programs that give an LLM powerful tools (like hacking access) without sufficient guardrails or human supervision.

Fidelity note

This summary is grounded in the provided transcript. The distinction between "Rationalist" philosophy and "Technical" AI safety is the core of the conversation. The summary accurately reflects the speakers' skepticism toward the current "Safety" establishment. The transcript does not provide a full list of every "Rationalist" figure, but focuses heavily on Eliezer Yudkowsky and the influence of the 1990s transhumanist movement.