Run an agent

Launch a task or dataset with an explicit agent and scoring method, then inspect the result.

Use synth run start to execute an agent on tasks. By default each task’s own verifier assesses its result. You can replace that verifier with a judge profile, or collect unscored outputs.

Before you start

• Install and sign in.
• Select an uploaded task or dataset you can access.
• Prepare a validated agent profile, or deliberately use the platform default.
• Check the task’s timeouts and resource requirements. A submission can incur compute costs; a dry run does not launch trials.

Use your namespace in place of acme. Task and dataset references below are examples: replace them with exact versions returned by your own Registry commands.

Run one task

synth run start \
  --task acme/review-permissions@v1 \
  --actor @reviewer.json \
  --dry-run

Read what the dry run checked and what remains unresolved. A successful dry run does not prove that the model will execute or that the task’s verifier is good.

When the request matches your intent, submit the same command without --dry-run:

synth run start \
  --task acme/review-permissions@v1 \
  --actor @reviewer.json

Copy the returned run ID. --task reuses or creates a private one-task dataset for that task; the run still has a pinned task set.

Run a dataset and repeat each task

Save eval-plan.json:

{"trials_per_task": 4}

synth run start \
  --dataset acme/review-evaluation@v1 \
  --actor @reviewer.json \
  --plan @eval-plan.json

A dataset with 20 entries and four trials per entry requests 80 trial slots. Repeated dataset members remain repeated entries. This is a work amount, not a promise that 80 sandboxes will run simultaneously.

Without --actor, the platform selects its default actor and the receipt identifies that choice. Use an explicit profile when reproducibility matters.

Choose the scoring method

Intent	Flags	Result
Use the task’s assessment	Omit --judge and --no-judge	The task verifier scores each trial.
Supply another assessor	--judge @judge.json or a pinned agent:// reference	That judge replaces the task verifier.
Collect outputs only	--no-judge	No verifier, judge or reward.

Omitting --judge does not disable scoring. Do not combine --judge and --no-judge. Training is a separate operation and cannot be unscored.

Inspect the work

synth run show <run-id>
synth run events <run-id> --follow --timeout 10m
synth run result <run-id>
synth run trials <run-id>

Stopping the local watch does not stop the remote run. A terminal execution state and task quality are different: read the output, score coverage and failure information before deciding what the result establishes.

To stop remote work deliberately:

synth run cancel <run-id> --yes

Cancellation is cooperative. Continue inspecting the run until it reports its terminal outcome and any cleanup limitation.

Read runs and evidence, then compare setups or train the model.