Train and test a checkpoint

Run a bounded training plan, verify the saved model, and compare it with the baseline on held-out tasks.

Training gives the model practice on scored tasks. The useful outcome is not merely a completed run: it is a saved model that you can evaluate under the same conditions as the baseline.

Prepare the experiment

You need an accessible practice dataset, a separate held-out dataset, and a validated actor profile selecting a supported trainable model. Keep a baseline copy of that profile and freeze the task verifier or judge used for comparison.

The profile supplies the model and execution recipe. Do not add separate model or harness flags to the training command.

Task resource limits and timeouts remain the defaults unless you explicitly override supported fields in the plan. Limits and resources distinguishes workload size from concurrency and resource policy.

Choose the work amount

Save this as train-plan.json:

{
  "num_batches": 3,
  "batch": {
    "tasks": 2,
    "trials_per_task": 4
  }
}

This requests 3 × 2 × 4 = 24 planned trial slots, in three batches. It does not guarantee six distinct tasks, three optimizer updates or three checkpoints. Inspect the selected tasks and actual outputs rather than assuming coverage from multiplication alone.

Training is scored. With no --judge, each task’s verifier supplies the score. An explicit judge profile replaces that verifier. There is no unscored training.

Inspect, then submit

synth train start \
  --dataset acme/review-practice@v1 \
  --actor @baseline.json \
  --plan @train-plan.json \
  --dry-run

Check the work amount, optimizer initialization and overrides. A dry run is not an execution test. To submit, remove --dry-run:

synth train start \
  --dataset acme/review-practice@v1 \
  --actor @baseline.json \
  --plan @train-plan.json

Keep the returned run ID. Use synth run to inspect training; there is no separate set of training readers.

Verify what training produced

synth run show <training-run-id>
synth run events <training-run-id> --follow --timeout 10m
synth run updates <training-run-id>
synth run checkpoints <training-run-id>
synth run result <training-run-id>

A batch can finish without a trainable update. A failed run can still have saved checkpoints. Read the actual update and checkpoint records, their scope and the terminal result.

Do not call a progress event, intended step or recovery-state file a saved model. Copy the full checkpoint UUID from the checkpoint list and inspect it:

synth checkpoint show <checkpoint-id>

Evaluate the checkpoint with the same recipe

Copy baseline.json to trained.json. Change only agent.model to the checkpoint UUID you selected. Keep the engine version, instructions, files, tools and output contract unchanged.

synth agent validate trained.json
synth run start \
  --dataset acme/review-held-out@v1 \
  --actor @trained.json \
  --plan @eval-plan.json

Run the baseline on the same held-out dataset and evaluation plan if you have not already done so. Use the same scoring method for both. See Evaluate and compare for eval-plan.json and the complete comparison.

Look for actual quality gains, regressions, false positives, reliability and execution cost. Training rewards come from changing models; they are not a replacement for this fresh fixed-checkpoint evaluation.

Starting from a checkpoint versus resuming optimization

Training starts with fresh optimizer state by default, including when the actor selects an existing checkpoint.

To request continuation of compatible optimizer and trainer state, set "optimizer_initialization": "resume" in the plan. The actor’s model must be a checkpoint UUID. A new submission still has its own run ID and work plan; the input checkpoint is not a checkpoint created by that new run.

A platform recovery within one run is different again. It should not be treated as an operator request for another training experiment.

Use a result deliberately

Retain the full selected profile, checkpoint identity, baseline and comparison results. The selected checkpoint can be used by a new profile; publishing that profile does not itself prove improvement or automatically replace an agent in your workflow.

If the comparison is inconclusive, keep the baseline. The improvement lifecycle explains how to choose the next experiment.