Improve your agent

Turn a recurring failure into a controlled experiment, then check whether the improvement holds on different work.

A useful improvement solves a problem you care about without introducing a worse one. Begin with a specific failure—such as a reviewer missing authorization bugs—not a target score alone.

Start with a baseline

Choose representative tasks and define success before changing the agent. Keep the task versions, agent profile, scoring rules and results together.

Inspect the actual work. If a reviewer reports no bugs, was the change correct, did the reviewer miss a real problem, or did the task omit the information needed to identify it? Those require different interventions.

Reserve held-out tasks now, before using results to choose a change. Keep nearby variants of the same example together so that a nearly identical practice task does not leak into the evaluation set.

Choose the intervention that fits the failure

What you observe	What to investigate
The agent cannot find a necessary file or tool	Task context, environment and harness setup.
The instruction permits incompatible interpretations	The task instruction and success criteria.
The agent spends its time on irrelevant work	Its instructions, tools and execution recipe.
The agent repeatedly makes a relevant reasoning mistake	The model’s capability, and whether practice or another model helps.
Good work receives a bad score	The assessment, before optimizing against it.

Editing a prompt or tool configuration changes the harness. Training changes model parameters. Both can affect quality, but neither guarantees improvement. Do not change the evaluator at the same time and then attribute a higher score to the agent.

Compare one meaningful difference

Use the same task versions, scoring rules and execution conditions for the baseline and comparison. Evaluate and compare walks through that process.

Repeat trials when behavior varies. Report the number attempted, the number with recorded scores, failures, timeouts and the actual work produced. Do not silently remove difficult tasks or substitute zero for a missing assessment.

If several things must change together, describe the result as a comparison of the complete setups. It does not isolate the effect of one component.

Test on held-out work

Run the baseline and changed setup on the reserved tasks with the same scoring rules. Look beyond the average:

• Did it solve the failures that motivated the change?
• Did it introduce false positives or regressions on previously good work?
• Is the result consistent across tasks and repeated trials?
• Does the actual output agree with the assessment?
• Are execution time and resource use acceptable for the intended workflow?

A higher practice score is useful diagnostic evidence, not proof of transfer. A completed optimization step or saved checkpoint establishes that an operation happened, not that the resulting agent is better.

Decide what to use

Adopt the changed setup when the evidence supports the intended use. Keep its exact profile and model selection, task and dataset versions, scoring rules, baseline results, comparison results and known limitations.

Keep the previous setup available. An inconclusive or negative result is a reason to investigate the next intervention, not a reason to relabel success.

Start with one task, then use evaluation and comparison to establish your baseline.