> ## Documentation Index
> Fetch the complete documentation index at: https://synthlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a test suite

> Keep the work fixed while you compare agents, and reserve independent tasks for the final exam.

A dataset is an ordered collection of exact task versions. It makes “test the same work again” a reproducible operation instead of a list you reconstruct from memory.

## Create a suite

Replace `acme` and the task names with your own references:

```bash theme={null}
synth dataset create review-practice \
  --task acme/review-permissions@v1 \
  --task acme/review-pagination@v2 \
  --visibility private --json
```

The response supplies the dataset's versioned reference. Inspect its actual membership before spending execution budget:

```bash theme={null}
synth dataset show acme/review-practice@v1 --json
synth dataset tasks acme/review-practice@v1 --json
```

Membership is ordered; repeated entries are not automatically evidence of distinct work. Count both entries and unique tasks when designing an experiment.

## Run the suite

An evaluation plan controls repetitions per task. Save this as `eval-plan.json`:

```json theme={null}
{"trials_per_task": 4}
```

```bash theme={null}
synth run start --dataset acme/review-practice@v1 \
  --actor @reviewer.json --plan @eval-plan.json --dry-run --json
```

Remove `--dry-run` to submit. Two task entries with four repetitions request eight trials; this does not prescribe eight simultaneous sandboxes. [Run an agent](/guides/managed-execution) explains scoring and progress.

## Keep practice separate from the exam

Use different datasets for:

* **Practice:** tasks used to adjust instructions, tools, or model weights.
* **In-distribution evaluation:** reserved tasks resembling practice work.
* **Future evaluation:** newer work, or a later time window, that tests whether the improvement carries forward.

Different task IDs do not guarantee independence. Check for the same pull request, duplicated source, shared answers and closely related changes across splits. Freeze membership and scoring before inspecting comparison results.

## Change and inspect a suite

Changing membership creates a new version; old runs keep their original inputs. Keep visibility private unless sharing is deliberate.

```bash theme={null}
synth dataset list --json
synth dataset runs acme/review-practice@v1 --json
```

For file-based membership, `synth dataset create --help` describes `--from`. Use [evaluation](/guides/evaluate-and-compare) to compare agents and [training](/guides/managed-training) to learn from the practice split.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.