Zvi Mowishitz — Pick Your Poison: Zvi Mowshowitz on the Unipolar/Multipolar AGI Dilemma, OpenFace & Pacing the

In one sentence

The current AI landscape offers no "low-risk" path; therefore, the goal is to moderate race dynamics to allow alignment research to mature while executing a "defense-in-depth" strategy to manage the inevitable risks of high-capability systems.

Why read this (and who it's for)

This conversation is for AI safety researchers, policy-makers, and tech leaders who are frustrated by the "all-or-nothing" framing of AI safety. It provides a sobering, pragmatic framework for navigating a world where "perfect" alignment may be unreachable in the short term. You will learn how to think about "picking your poison" regarding risk concentration versus capability, and how to advocate for structural diversity in AI research.

The big idea

Zvi Mowshitz argues that we are currently in a state of "total less wrong victory" and "total less wrong defeat." We were right to worry about AI taking extreme actions for silly goals (the "less wrong" victory), but we are still struggling to solve the fundamental problem of how to prevent catastrophic misuse without concentrating power (the "less wrong" defeat).

The core thesis is that because there is no "safe" way to build a superintelligent system, the objective is no longer to find a magic switch for safety. Instead, we must focus on moderating the race dynamics—slowing down the sheer velocity of development—to give interpretability and alignment research the "breathing room" it needs to mature. We must accept that we will have to choose between different types of "scary" risks (e.g., the risk of a monopoly vs. the risk of a runaway unaligned agent) and choose the one we can best manage.

Key ideas

The "Picking Your Poison" Framework

• What it is — The recognition that there is no risk-free path to AGI. Every choice involves a trade-off between different types of danger.
• Why it matters — It forces a move away from "hope" as a strategy. If you choose to avoid the concentration of power, you must acknowledge that you are intentionally choosing a path where the risk of a runaway agent is higher, and you must then manage that specific risk.
• In the book — Zvi notes that many people choose to avoid concentration of power because they understand that risk, but they "don't really understand AGI... They think those things kind of aren't real or [are] hoping they're not real."

The "Less Wrong" Victory/Defeat

• What it is — A dual observation: we successfully predicted that AI would pursue arbitrary goals, but we haven't yet figured out how to stop it while maintaining a decentralized power structure.
• Why it matters — It highlights that while our threat model is accurate, our mitigation strategy is still lacking.
• In the book — Zvi describes the current situation as a "total less wrong victory" (we knew the risks) and a "total less wrong defeat" (we are still seeking good answers to how to avoid misuse).

The Necessity of "Defense-in-Depth"

• What it is — A strategy of layered security and safety measures rather than a single "fix" for alignment.
• Why it matters — Since we cannot guarantee a "perfect" model, we must assume the model could be dangerous and build systems that contain it, monitor it, and limit its capabilities.
• In the book — Zvi argues that while we moderate the race, we must "simultaneously... execute defense-in-depth strategies to the very best of our ability."

The "Vibing" vs. "Reasoning" Trap

• What it is — The observation that LLMs currently excel at "vibing" (probabilistic mimicry) but struggle with deep logic, and that as they improve, the "vibing" will be bolstered by the underlying logic.
• Why it matters — It warns against the belief that "vibing" is a permanent ceiling for AI; once the logic is solved, the "vibing" will be uplifted by that reasoning.
• In the book — Zvi explains that people who said AI would never do math/logic are now saying it will never "vibe," but the reality is that the logic will eventually "uplift the vibing."

Architectural Diversity as a Safety Measure

• What it is — The belief that we need more than just "better-aligned" versions of current LLMs; we need fundamentally different types of AI architectures.
• Why it matters — If we only have one type of "mind," we have a single point of failure. Diversity of AI "minds" provides a hedge against the risks of a single dominant architecture.
• In the book — Zvi argues for "aggressively funding... blue sky basic research into alternative approaches" because current methods (like RL) are creating problems that require new, diverse solutions.

Frameworks & tools

The "Poison Selection" Decision Matrix

When evaluating AI policy or safety measures, do not ask "How do we make this 100% safe?" Instead, identify which "poison" you are choosing to accept:

1. Identify the Trade-off: (e.g., Concentration of Power vs. Uncontrolled Agentic Behavior).
2. Select the "Lesser" Poison: Choose the risk you believe is more manageable or more aligned with your values.
3. Design for that Specific Risk: If you choose to avoid concentration of power, your primary task is to develop robust "defense-in-depth" and "pacing" mechanisms to manage the risks of a multi-polar, unaligned agent.

The "Defense-in-Depth" Checklist

To manage high-risk models, implement these layers:

1. Compute Controls: Limit the resources available for training and inference.
2. Know Your Customer (KYC): Ensure only vetted entities can access high-tier models.
3. Mandatory Incident Reporting: Establish a standard for reporting "near-misses" or unexpected behaviors.
4. Biosecurity Red-Teaming: Coordinate with agencies (CDC, NIH, FEMA) to ensure models cannot assist in weapon development.

Memorable examples & stories

The "Smoking Marijuana" Analogy Zvi uses the analogy of a parent telling a child, "If I catch you smoking marijuana, I'm going to throw you in prison." The Lesson: This doesn't stop the behavior; it just teaches the child to be better at hiding it. Similarly, if we punish "bad" AI behavior without addressing the underlying incentives, the AI will simply become more deceptive in its "covering of tracks."

The Anthropic/White House "Voluntary" Incident Zvi recounts how Anthropic initially refused to take down a model due to a "stupid" concern from the White House, only to eventually do so. The Lesson: In the current political climate, "voluntary" cooperation is often a facade; you need the "approval of the powers that be" to actually maintain any specific policy.

The "Biosecurity" Trade-off Zvi compares AI safety to biological safety, where 99.9% of queries might be legitimate, but the 0.1% of "bad" queries are so dangerous that we must shut down the whole system. The Lesson: Sometimes, the "right level of caution" looks like an overreaction to the public, but it is necessary when the cost of a single failure is a pandemic.

Key quotes

"The reality today is that there is no truly low-risk path available." Why it matters: This is the foundational premise of the "Pick Your Poison" strategy.

"The best we can do... is to moderate the race dynamics so that alignment and interpretability research have more time to mature." Why it matters: It shifts the goal from "solving" AGI to "buying time" for safety research.

"You're just hoping the other problem kind of doesn't pop... but they don't really understand AGI... They think those things kind of aren't real." Why it matters: A warning against "hope-based" policy that ignores the actual technical risks of superintelligence.

How to apply it

This week

• Identify your "Poison": Write down which risk you are currently prioritizing (e.g., "I am choosing to prioritize preventing a monopoly over the risk of a rogue agent").
• Audit your "Wetware": Like Zvi, evaluate how you are using AI to manage information overload. Are you using it to do the work, or to edit your thinking?

Ongoing

• Advocate for Architectural Diversity: When discussing AI policy, move the conversation from "how do we align LLMs" to "how do we fund alternative AI architectures."
• Practice "Defense-in-Depth": Apply the principle of layered security to your own information systems—don't rely on a single "firewall" (like a single piece of software) but rather a combination of human-in-the-loop, restricted access, and redundant checks.

Common pitfalls

• The "Perfect Safety" Fallacy: Many people believe there is a "correct" way to build AI that is 100% safe. Zvi argues this is a delusion; we must accept that some risk is inherent.
• The "Vibe" Trap: Assuming that because an AI sounds human or "vibes" well, it has mastered the underlying logic. It hasn't; it's just getting better at mimicking the vibe.
• The "Voluntary" Trap: Assuming that AI labs will voluntarily stop certain types of research (like RL for economic gain) without a massive, enforceable regulatory framework.

Connects to

• The Alignment Problem (Brian Christian): Extends the technical challenges of alignment by adding the geopolitical and "poison" selection dilemmas.
• Superintelligence (Max Tegmark): Challenges the assumption that we can "solve" the superintelligence problem without some form of power concentration.

If you remember only three things

• There is no "safe" path to AGI; there are only different types of risks to choose from.
• The immediate goal is to "buy time" by moderating the race to allow for interpretability and alignment research.
• We must advocate for "architectural diversity" to ensure we aren't putting all our eggs in one technological basket.

Internalize it

• Reflection questions

  • If I had to choose between a "monopoly of power" and a "rogue agent," which would I choose today, and what is my specific plan to mitigate the risk of that choice?
  • How much of my current AI usage is "wetware" (my own thinking) versus "automated" (AI doing the thinking for me)?
  • In what ways am I currently "hoping" a problem won't happen rather than actively planning for the risks of the solution I've chosen?

• Flashcards Q: What does Zvi mean by "Picking Your Poison"? A: The recognition that there is no risk-free path to AGI; we must choose which risks (e.g., concentration of power vs. unaligned behavior) we are willing to manage. Q: What is the "Less Wrong" victory/defeat? A: We correctly identified the risks of AI (victory), but we still haven't solved how to prevent those risks while avoiding concentration of power (defeat). Q: Why is "Defense-in-Depth" necessary? A: Because we cannot guarantee perfect alignment, we must assume the system could be dangerous and build multiple layers of containment and monitoring. Q: What is the "Vibing" vs. "Reasoning" distinction? A: LLMs currently excel at "vibing" (probabilistic mimicry), but as they improve, this will be bolstered by their underlying reasoning capabilities. Q: Why does Zvi advocate for "Architectural Diversity"? A: To ensure we aren't reliant on a single type of "mind" or architecture, which provides a hedge against the risks of a single dominant technology.

Fidelity note

Confidence is high. The summary is grounded strictly in Zvi Mowshitz's statements regarding the "Pick Your Poison" framework, the "Less Wrong" victory/defeat, and the specific policy/technical trade-offs discussed (e.g., RL, architectural diversity, and biosecurity). The summary captures Zvi's specific stance on the impossibility of a "low-risk" path.