Build a test suite

Keep the work fixed while you compare agents, and reserve independent tasks for the final exam.

A dataset is an ordered collection of exact task versions. It makes “test the same work again” a reproducible operation instead of a list you reconstruct from memory.

Create a suite

Replace acme and the task names with your own references:

synth dataset create review-practice \
  --task acme/review-permissions@v1 \
  --task acme/review-pagination@v2 \
  --visibility private --json

The response supplies the dataset’s versioned reference. Inspect its actual membership before spending execution budget:

synth dataset show acme/review-practice@v1 --json
synth dataset tasks acme/review-practice@v1 --json

Membership is ordered; repeated entries are not automatically evidence of distinct work. Count both entries and unique tasks when designing an experiment.

Run the suite

An evaluation plan controls repetitions per task. Save this as eval-plan.json:

{"trials_per_task": 4}

synth run start --dataset acme/review-practice@v1 \
  --actor @reviewer.json --plan @eval-plan.json --dry-run --json

Remove --dry-run to submit. Two task entries with four repetitions request eight trials; this does not prescribe eight simultaneous sandboxes. Run an agent explains scoring and progress.

Keep practice separate from the exam

Use different datasets for:

• Practice: tasks used to adjust instructions, tools, or model weights.
• In-distribution evaluation: reserved tasks resembling practice work.
• Future evaluation: newer work, or a later time window, that tests whether the improvement carries forward.

Different task IDs do not guarantee independence. Check for the same pull request, duplicated source, shared answers and closely related changes across splits. Freeze membership and scoring before inspecting comparison results.

Change and inspect a suite

Changing membership creates a new version; old runs keep their original inputs. Keep visibility private unless sharing is deliberate.

synth dataset list --json
synth dataset runs acme/review-practice@v1 --json

For file-based membership, synth dataset create --help describes --from. Use evaluation to compare agents and training to learn from the practice split.