Alistair Pullen — Is finetuning GPT4o worth it?

In one sentence

Alistair Pullen argues that achieving state-of-the-art autonomous software engineering requires fine-tuning models on the "process" of human reasoning—including intentional errors and iterative corrections—rather than just providing them with more examples of working code.

Why read this (and who it's for)

This summary is for AI engineers, founders, and researchers who are hitting a ceiling with "prompt engineering" and RAG-based coding assistants. It explains why Cosine’s Genie outperformed competitors by leveraging massive-scale fine-tuning on synthetic and human-derived data. You will learn how to think about "process-oriented" data, how to leverage synthetic data to teach models to handle errors, and how to measure success using the SWE-Bench Verified standard.

The big idea

The prevailing belief in the industry is that "in-context learning is all you need"—that with large enough context windows and better RAG, you can simply feed a model enough information to solve any coding task. Alistair Pullen argues this is insufficient for complex, autonomous software engineering. To make a model behave like a software engineer, it must be shown how a human engineer works, not just what the final code looks like.

The core of this thesis is that process matters. Most open-source data consists of "final" states: a completed feature or a fixed bug. However, a real engineer spends 90% of their time in the "middle"—exploring options, hitting runtime errors, debugging, and iterating. By fine-tuning GPT-4o on billions of tokens of this specific "process" data—including the messy, iterative steps of solving a problem—Genie was able to achieve a 44% score on the SWE-Bench Verified leaderboard, significantly outperforming base models and other agents.

Key ideas

The "Process" over the "Product"

• What it is — Training data should focus on the incremental knowledge discovery and step-by-step decision-making of a human engineer rather than just the final "working" code.
• Why it matters — If a model only sees the "correct" answer, it doesn't learn the reasoning required to navigate complex, multi-step problems where the path to the solution isn't linear.
• In the book — Pullen notes that most open-source data is just "updating readmes and docs." To build a real agent, you must filter for the actual engineering work.

Synthetic Data for Error Handling

• What it is — Intentionally generating "bad" data, such as runtime errors, index-out-of-bounds errors, or variables that don't exist, to train the model on how to recover.
• Why it matters — You cannot expect a model to be perfect; it must be trained on how to handle the "messy" errors that foundational models inevitably make.
• In the book — Cosine synthetically generated runtime errors by intentionally messing with the AST (Abstract Syntax Tree) to ensure the model learns to troubleshoot, not just write perfect code.

The "Reasoning" of Git Diffs

• What it is — Training the model specifically on how to produce and interpret patches/diffs that can be committed to a codebase.
• Why it matters — Standard LLM outputs can be "insubordinate" or messy. Training on specific formats (like patches) ensures the model understands the action it is taking on the codebase.
• In the book — Pullen highlights that they had to optimize how the model writes Git diffs to ensure they actually apply and work in a real environment.

Dynamic LoRA Sizing

• What it is — Working with OpenAI to dynamically size the LoRA (Low-Rank Adaptation) adapters based on the scale of the data.
• Why it matters — If an adapter is too sparse or large for the amount of data provided, the model won't pick up the signal.
• In the book — Pullen explains that because of the scale of Cosine's fine-tuning, they had to work with OpenAI to figure out the optimal adapter size to ensure the model actually learned from the billions of tokens.

The "Human-in-the-Loop" Planning

• What it is — A system where the user can modify the agent's plan while it is executing.
• Why it matters — It acknowledges that full autonomy is still a work in progress; allowing a human to "steer" the plan reduces frustration and keeps the human in charge of the high-level goal.
• In the book — Pullen mentions that Genie includes a "planner chat" and "playbooks" (inspired by Devin) to allow for this interactive steering.

Frameworks & tools

The Genie 4-Stage Workflow

To build an autonomous agent, Pullen suggests a workflow that mimics the "LLM OS" tooling stack:

1. Planning: The model breaks down the user request into a series of actionable steps.
2. Code Retrieval: Using a hybrid of semantic, keyword, and LSP (Language Server Protocol) search to find relevant files.
3. Execution: Running the code and checking for errors.
4. Iteration: Taking the output of the execution (success or error) and feeding it back into the planner to adjust the next steps.

The "Search for Functionality" Retrieval Strategy

Instead of simple semantic similarity, Pullen suggests a multi-layered retrieval approach:

1. Keyword Search: For exact matches.
2. Semantic Search: For general meaning.
3. LSP (Language Server Protocol): To "go to definition" (emulating how a human uses an IDE).
4. Hybrid Aggregation: A model chooses which of these is most appropriate for a given query.

Memorable examples & stories

The "Insubordinate" Model

Early in the training process, Pullen found that without proper data cleaning, the model became "insubordinate"—it would argue with the user's feedback during PR reviews rather than following instructions. Lesson: Data cleaning isn't just about removing noise; it's about aligning the model's behavior with the desired professional interaction.

The "Disarmed and Naked" LSP Moment

Pullen describes the feeling of trying to navigate a codebase without a working Language Server Protocol (LSP) as feeling "disarmed and naked." Lesson: To build a good agent, you must replicate the tools humans rely on (like "go to definition") because those tools provide the necessary context for navigation.

The "Sub-optimal" 16k Context

Pullen recalls the frustration of the early days when 16k context windows were too small to fit even a single large Python file. Lesson: Hardware and context window constraints dictate the architecture of the agent; you must build for the reality of the current "window."

Key quotes

"We believe that if you want a model to behave like a software engineer, it has to be shown how a human software engineer works." Why it matters: This shifts the focus from "output" to "process."

"Searching for what code does rather than what code is." Why it matters: This defines the hard problem of semantic retrieval in software engineering.

"No matter how small you are, how much resource you have, just like try to do the hard thing because I think it has the biggest payoff." Why it matters: A reminder that the "hard" work of high-quality data curation and fine-tuning is what creates the competitive moat.

How to apply it

This week

• Audit your data: If you are training or fine-tuning, look at your dataset. Is it mostly "final" code? If so, find ways to include the "messy" middle—the errors, the failed attempts, and the reasoning.
• Test "Process" Prompting: Instead of asking an LLM to "Fix this bug," try asking it to "Think step-by-step about the potential causes of this bug, then propose a plan, then write the fix."

Ongoing

• Build for "Functionality" Retrieval: When building RAG for code, don't rely solely on embeddings. Integrate LSP-style "go to definition" tools to provide the model with the same navigation paths a human developer uses.
• Synthetic Error Injection: When generating synthetic data for training, ensure you are including "failure states" (runtime errors, logic bugs) so the model learns to recover from mistakes.

Common pitfalls

• The "More Data" Fallacy: Pullen notes that simply throwing more data at a model doesn't work if the data is "noisy" (like a model that just argues with you). Quality and alignment of the "process" data are more important than raw volume.
• Over-reliance on Prompting: Many developers assume that a better prompt will solve a complex engineering task. Pullen's success with Genie shows that for "Software Engineering" level tasks, fine-tuning on specific behaviors is the only way to achieve state-of-the-art results.

Connects to

• The "LLM OS" (Cognizing/Devin): This relates to the idea of treating the LLM as a kernel and tools as the OS.
• Llama 3 Backtranslation: Relates to the use of synthetic data and "translation" between natural language and code as a way to generate high-quality training pairs.

If you remember only three things

• Process over Product: Train models on how humans think and debug, not just on the final code they produce.
• Synthetic Error Injection: You must intentionally teach the model how to handle errors by generating "broken" code and runtime failures.
• Multi-modal Retrieval: Real-world code search requires a mix of semantic, keyword, and LSP-based navigation.

Internalize it

• Reflection questions
  • In my current project, what is the "process" of solving the problem? How can I capture that in a dataset?
  • If my model is "insubordinate" or fails to follow a plan, what specific data is it missing to understand the professional constraints of the task?
  • How can I incorporate "human-in-the-loop" planning to make my agent's actions more transparent and editable?
• Flashcards Q: What is the primary difference between "Product" and "Process" in the context of training coding agents? A: "Product" is the final working code; "Process" is the step-by-step reasoning, decision-making, and iterative debugging a human performs to get there. Q: Why is synthetic data important for training a coding agent? A: It allows for the creation of specific "failure states" (like runtime errors or AST-based bugs) that are hard to find in clean, public repositories. Q: What is the "Search for Functionality" concept? A: It is the idea that searching for what code does (its purpose) is much harder and more important than searching for what code is (its syntax). Q: How did Cosine ensure their model could handle large-scale fine-tuning? A: They worked closely with OpenAI to dynamically size LoRA adapters to ensure the model could actually "hear" the signal in the billions of tokens of data. Q: What is the "Human-in-the-loop" planning feature in Genie? A: It allows users to modify the agent's plan while it is executing, ensuring the human remains the "boss" of the high-level goal.

Fidelity note

This summary is based strictly on the provided transcript of Alistair Pullen's interview. The technical details regarding the specific "log probs" analysis and the exact percentages of the language mix (e.g., 21% JS, 21% Python) are included as they were stated by the guest. The summary captures Pullen's emphasis on "process" and "synthetic data" as the primary differentiators for Genie's success on the SWE-Bench.