Tasks

Create, version, share, run, and inspect the work your agents need to master.

A task turns one piece of real work into a repeatable test. It keeps the instruction, environment, verifier, and exact version together so you can run the same work across agent setups and trust the comparison.

Publish Harbor-compatible tasks, run exact versions in the cloud, and inspect their results in the Dashboard.

Create it once

Package the work, environment, and success criteria in one portable directory.

Run any setup

Keep the task fixed while you compare supported agents and models.

Keep every result

Attach runs, scores, failures, trajectories, and artifacts to the exact task version.

What a task contains

Part	What it does
Instruction	Tells the agent what outcome to produce.
Environment	Provides the files, tools, services, and resource bounds it needs.
Verifier	Checks the result and returns a useful score or explanation.
Runner policy	Declares timeouts and Harbor-compatible execution limits.

Synth uses the Harbor task format. The same task directory can run locally with Harbor and in Synth cloud without a second definition.

To use a task that your team has already prepared, start with Run your first task. Local and managed execution still need compatible images, tools and resource settings.

Make the work realistic

Give the actor enough context to investigate: surrounding source, callers, contracts and relevant documentation, not merely the line containing a known answer. Organize abundant context so navigation is part of the job. The task’s instruction should state the work and output contract, not disclose the result.

Keep private reference answers and scoring code out of actor-visible files and images. Copy only the intended workspace into the environment; copying the whole task directory can expose the verifier. The assessor needs enough evidence to justify its decision too. Judging explains that separate role.

Create a task

Before using the commands below, install the Synth CLI and sign in.

Start with a directory that keeps the task definition at its root:

code-review/
├── task.toml
├── instruction.md
├── environment/
└── tests/

For managed execution, task.toml must declare a positive agent timeout plus explicit CPU and memory bounds:

[agent]
timeout_sec = 600

[environment]
cpus = 1
memory_mb = 4096

Inspect the exact package before uploading anything:

synth task upload ./code-review \
  --name code-review \
  --visibility private \
  --dry-run \
  --json

When the package looks right, publish its first immutable version:

synth task upload ./code-review \
  --name code-review \
  --visibility private \
  --json

The output identifies the task and its new version. Keep that version when you want to repeat the same work.

The first accepted upload creates v1. Uploading changed packaged content under the same name creates v2, then v3. Retrying identical content reuses the existing immutable version and returns deduplicated: true.

Check the task directory for credentials, private transcripts, unrelated workspace files, and verifier code you would not want a runner to execute. The dry run packages and inspects locally without creating a Registry version.

Find your tasks

List tasks available in your selected organization:

synth task list --json

Search your private and organization-visible tasks:

synth task list \
  --query code-review \
  --visibility organization \
  --json

Add --all-accessible when you also want public tasks owned by other organizations. Customer resources remain private unless you deliberately change their visibility.

Inspect a task and its versions

A bare name resolves the current version in your selected organization. Use @vN or a digest when the exact version matters:

synth task show acme/code-review@v1

Download and verify an exact version into a new directory:

synth task download acme/code-review@v1 \
  --output ./code-review-v1 \
  --json

The CLI verifies the Registry digest before atomically unpacking the task. Add --archive when you want the verified package file instead.

Update details or visibility

Task content is immutable. Edit the local directory and upload it again to create the next version; existing runs continue to point at the old bytes.

Display name, description, and visibility belong to the logical task and can change without rewriting version history.

Open the task in the Task Registry and select Edit to review its details and visibility. For an automated update, use the API reference.

Use private visibility for owner-only work, organization visibility for teammates, and public visibility only when anyone should be able to discover it. The CLI accepts --visibility private, --visibility organization, or --visibility public. Publishing is deliberate and auditable.

Run an exact version

Cloud executions use the exact task version you select. This keeps every score and artifact comparable even after you publish a newer version.

Run in Synth cloud

Run Claude Code and Codex against the exact version and compare their scores.

Test locally with Harbor

Check the task and verifier locally before spending cloud runtime.

Review every run

Inspect groups and trials attached to the exact task directly from the CLI:

synth task groups acme/code-review@v1 --links
synth task trials acme/code-review@v1 --links

Open the Task Registry to see versions and executions together. Registry matrix runs open in the managed-run view; Harbor jobs link to the native Harbor Viewer for trials, trajectories, verifier output, and artifacts.

Build a test suite

Group exact task versions so every setup is tested on the same work.