Aman Sanger — ICLR 2024 — Best Papers & Talks — ft. Graham Neubig

In one sentence

To achieve reliable AI agents, we must move beyond "outcome-only" evaluation and training toward "process-supervised" reasoning, multi-step planning, and benchmarks that mirror the messy, multi-file complexity of real-world software engineering and web navigation.

Why read this (and who it's for)

This summary distills the core technical advancements from ICLR 2024 regarding agentic behavior and reasoning. It is for AI engineers, researchers, and product managers who need to understand why current LLMs struggle with complex, multi-step tasks (like coding a full feature) and what specific architectural shifts—such as Self-RAG, Process Supervision, and Meta-Programming—are being deployed to solve these issues. You will learn how to evaluate model "thinking" rather than just "answering."

The big idea

The central thesis of the ICLR 2024 highlights is that LLMs are transitioning from "Chatbots" to "Agents." While early LLM success was measured by the ability to generate a correct-looking response to a prompt, the new frontier is Agency: the ability to plan, use tools, navigate the web, and manage complex codebases.

However, this transition reveals a "capability gap." Models often fail at tasks that require long-term planning or multi-step reasoning because they lack a "process" for thinking. To bridge this, the research community is moving toward Process Supervision (rewarding the correct steps of a math problem, not just the final answer) and Self-Reflection (where the model critiques its own retrieved information). Furthermore, because "easy" benchmarks are now saturated, the new standard for success is Real-World Complexity—specifically, can the model resolve a GitHub issue across multiple files or navigate a live website without human intervention?

Key ideas

The Shift to Software Engineering (SWE) Benchmarks

• What it is — Moving from "Write a Python function" to "Fix this bug in a repository of 50 files."
• Why it matters — Traditional benchmarks are too easy; they don't require the model to coordinate changes across different classes or functions.
• In the book — The SWE-bench (ICLR Oral) evaluates models on 2,294 real-world GitHub issues. Currently, even top models like Claude 2 only solve ~1.96% of these, highlighting how much harder "real" work is compared to "synthetic" tasks.

Process Supervision vs. Outcome Supervision

• What it is — Training a model by rewarding the correct intermediate reasoning steps rather than just the final result.
• Why it matters — Outcome supervision (e.g., "Is the answer 42?") doesn't tell the model why it was right, leading to "hallucinated" logic. Process supervision ensures the model learns a reliable path to the answer.
• In the book — The "Let’s Verify Step By Step" paper shows that process-supervised models significantly outperform outcome-supervised ones on the challenging MATH dataset.

Self-Reflective Retrieval (Self-RAG)

• What it is — A framework where an LLM learns to retrieve information, generate a response, and then critique that response using "reflection tokens."
• Why it matters — Standard RAG often retrieves irrelevant info and stuffs it into the prompt. Self-RAG makes the model "aware" of whether the retrieved data is actually useful.
• In the book — Self-RAG (7B/13B) outperforms ChatGPT on fact-verification and reasoning by selectively choosing when to retrieve and how to reflect on the results.

The "Reversal Curse" in LLMs

• What it is — The discovery that if a model learns "A is B," it does not automatically learn "B is A."
• Why it matters — It exposes a fundamental limitation in how auto-regressive models generalize; they learn associations, not underlying logical truths.
• In the book — The study shows that even GPT-4 fails to reverse a fact (e.g., knowing who a celebrity's mother is but not who her son is) 67% of the time.

Meta-Programming for Multi-Agent Systems

• What it is — Using "Standardized Operating Procedures" (SOPs) to coordinate multiple agents in an assembly-line fashion.
• Why it matters — Complex tasks shouldn't be one big prompt; they should be broken into roles (e.g., Developer, Reviewer, Manager) that follow a human-like workflow.
• In the book — MetaGPT uses this to reduce errors by having agents verify intermediate results before moving to the next step in the "assembly line."

Frameworks & tools

The Process Supervision Framework

To train a model to reason reliably, move from Outcome to Process:

1. Identify the Reasoning Path: Break down a complex problem (like a math proof) into discrete, logical steps.
2. Label the Steps: Provide human feedback on each intermediate step (e.g., "Correct," "Incorrect," "Minor Error").
3. Train the Reward Model: Use these labels to train a model that predicts the probability of a correct step.
4. Iterative Refinement: Use the Reward Model to guide the LLM's generation of the next step.

The MetaGPT Assembly Line

To build a multi-agent system for complex tasks:

1. Define SOPs: Write down the human workflow for the task (e.g., "How do we write a software requirement document?").
2. Assign Roles: Create specific agents (e.g., Product Manager, Architect, Coder).
3. Sequential Execution: Ensure Agent A's output is the input for Agent B.
4. Verification: Require an agent to "verify" the intermediate result before the next agent begins.

Memorable examples & stories

The 1.96% Success Rate

The SWE-bench results serve as a "reality check" for the industry. While LLMs can write a single function, they struggle to navigate a real repository to fix a bug. Lesson: We are still in the early stages of "Agentic" AI; the gap between "code generation" and "software engineering" is massive.

The Reversal Curse

The study of "A is B" vs "B is A" shows that LLMs are essentially "associative" rather than "logical." Lesson: Don't assume that because a model knows a fact, it understands the relationship between the entities in that fact.

The "Bad" RAG Implementation

The authors of Self-RAG point out that many current RAG systems are "dumb"—they retrieve a fixed number of chunks and shove them into the prompt regardless of relevance. Lesson: Retrieval is only useful if the model can intelligently decide what to ignore.

Key quotes

"Process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset." (Paraphrased from Let’s Verify Step By Step) Why it matters: It shifts the goal from "getting the right answer" to "learning the right way to think."

"We introduce MetaGPT, an innovative meta-programming framework incorporating efficient human workflows into LLM-based multi-agent collaborations." (From MetaGPT) Why it matters: It argues that the secret to complex AI behavior is mimicking human organizational structures.

"The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation." (From The Reversal Curse) Why it matters: It highlights a fundamental architectural limitation in current LLM training.

How to apply it

This week

• Audit your RAG: If you are using RAG, check if your model is being forced to use irrelevant context. Try a "Self-Reflection" prompt where the model must state if the retrieved context is useful before answering.
• Switch to Process-Oriented Prompting: Instead of asking for a final answer, ask the model to "Think step-by-step" and verify each step before proceeding to the next.

Ongoing

• Use SOPs for Agents: When building multi-agent systems, don't just give them a goal. Give them a "Standard Operating Procedure" (SOP) and a specific role.
• Benchmark for Realism: When evaluating your own models, stop using "Trivia" questions. Start using "Action" tasks (e.g., "Go to this website and find X").

Common pitfalls

• The "Easy Benchmark" Trap: Many developers think their model is "smart" because it scores high on MMLU or GSM8K. These are often "memorized" or "contaminated" (as shown by the Proving Test Set Contamination paper).
• Over-reliance on Outcome Supervision: Training a model to give the right answer without ensuring it follows the right logic leads to brittle models that fail on slightly different prompts.
• Ignoring the Reversal Curse: Assuming that "knowledge" in an LLM is equivalent to "understanding" of a relationship.

Connects to

• DSPy: Connects to the idea of "compiling" these complex agentic pipelines into optimized graphs rather than hard-coded prompts.
• OpenDevin: Connects to the practical application of the SWE-bench and WebArena concepts in a real-world coding agent.

If you remember only three things

• Process > Outcome: To make models smarter, reward the steps of reasoning, not just the final answer.
• Agency = Planning + Tools: True agents require the ability to decompose a goal into sub-tasks, plan, and use external tools (like a web browser or a Python interpreter).
• The Reversal Curse: LLMs are associative, not inherently logical; they