Evaluate and compare agents

Measure a baseline and a deliberate change on the same tasks with the same assessment.

Evaluation should answer a decision: which setup is useful for this work, and what changed? A high number without a task set, scoring rule and baseline does not answer that question.

Freeze the comparison

Before submitting work, record:

• the exact dataset version;
• the baseline and comparison profiles;
• the task verifier or exact judge profile;
• trials per task, timeouts and any resource overrides; and
• the criteria you will use to decide whether a change helped.

Keep a held-out dataset separate from the examples used to tune instructions, choose a model or train. Avoid nearly identical tasks across the two sets.

Run the baseline

Prepare baseline.json using agent profiles. Save eval-plan.json with:

{"trials_per_task": 4}

Then submit:

synth run start \
  --dataset acme/review-held-out@v1 \
  --actor @baseline.json \
  --plan @eval-plan.json

The task verifier scores this run. If your experiment uses a judge profile, add the same pinned --judge selection to both runs.

Run the changed setup

Save the changed recipe as comparison.json and submit against the same dataset and plan:

synth run start \
  --dataset acme/review-held-out@v1 \
  --actor @comparison.json \
  --plan @eval-plan.json

These are two separate paid submissions. Save both IDs. If a submission’s outcome is uncertain, inspect it before retrying; see retry safety.

For a trained-model comparison, keep the recipe unchanged except for the model selector: base model in the baseline, saved checkpoint UUID in the comparison.

Read more than the mean

For each run, inspect:

synth run show <run-id>
synth run trials <run-id>

After each run reaches a terminal state, read and export its result:

synth run result <run-id>
synth run export <run-id> --output ./<run-id>-evidence

Substitute the same run ID in the command and output directory, so the two exports stay separate. The directory must not already exist. Compare the same tasks and report score coverage, failures and timeouts alongside reward. Missing scores are not automatically zero and should not silently disappear.

Read a sample of the actual work from both setups. For a reviewer, ask whether its findings are correct, consequential and actionable—not just whether the judge assigned more points. Include tasks with regressions or unexpected scores.

For a small test set, describe the result as preliminary. Repeated trials measure variation on the selected tasks; they do not create more independent tasks or prove broad generalization.

Reassess retained outputs without rerunning the actor

Use regrading when the question concerns the assessment itself:

synth run regrade <source-run-id> \
  --judge @judge.json \
  --dry-run

The dry run reads the available retained artifacts and describes the work. Remove --dry-run only when you intend to submit the new assessment.

A regrade uses retained actor output files, not a complete historical sandbox snapshot. Trials with missing or unreadable output archives can be excluded; inspect the reported coverage. The source task must support the reconstruction described by the command.

The new assessment does not replace the original scores and does not establish how the original actor would perform if run again. Keep the source and regrade IDs together. Do not silently compare one setup’s original scores with another setup’s scores from a different judge.

Continue with the improvement decision.