How Synth fits together

Understand the agent, the work, the assessment and the evidence before designing an experiment.

Synth is organized around a practical question: what would make an agent more useful at your work, and how would you know?

Start with representative work. Make it repeatable, establish how you will assess the result, and compare a baseline with a deliberate change. Preserve the evidence so that a promising result can be checked rather than remembered.

An agent is more than its model

Agent = Model + Harness + Context.

• The model supplies learned capability and judgment.
• The harness equips the model: instructions, tools, permissions, skills, retrieval and the mechanisms for managing its context.
• The context is the information available to the model at a particular point in its work.

The environment is external: the repository, files, processes and services in which the agent works. A task prepares that environment; it does not become part of the model.

For example, Claude Code is a harness. Using that harness with a supported open model does not mean you are using or training Anthropic’s model. Record the actual setup rather than treating the tool’s name as the model identity.

The pieces you work with

Piece	The question it answers
Task	What work should be attempted, with which starting files, environment and default verification?
Task version	Which immutable package of that work is selected?
Dataset	Which exact task versions belong to this ordered collection?
Agent profile	Which model and execution recipe should act?
Verifier or judge	How is the outcome assessed?
Plan	How much work should be performed, with which explicit execution settings?
Run	What was accepted and what happened during execution?
Checkpoint	Which saved model state did training produce or a later run consume?
Evidence	Which outputs, assessments, trajectories and events support the result?

A profile is not the task: it defines the agent’s recipe, while the task defines the work and environment. A checkpoint is not the whole agent: using it still requires an execution recipe and appropriate task context.

See profiles, tasks, and results for those boundaries in practice.

Execute, evaluate and train

Execution asks the agent to do the work and collects the output. It can be unscored.

Evaluation runs a fixed setup and records assessments of its results. By default, the task’s own verifier scores each trial. An explicit judge profile replaces that verifier; it is not required for every evaluation.

Training uses scored trials to optimize the model. The model can change between batches, so its training scores are not one fixed-model evaluation. Training cannot be unscored.

The CLI reflects these purposes:

• synth run start executes tasks, with scoring unless you request --no-judge.
• synth train start starts a scored optimization plan.
• synth run show and the other run readers inspect either kind.

There is one run history, not separate identities for the same work. Choose the right command.

Work, practice and held-out evaluation

Real work establishes relevance. Choose situations where better performance would help you or your team.

Practice supports improvement. Use representative tasks to investigate failures, choose changes and train.

Held-out evaluation tests the change. Reserve different examples before improvement work begins. Evaluate both the baseline and the changed agent under the same conditions. If you repeatedly tune against that evaluation set, it is no longer an untouched test.

A completed run, a high reward and a genuinely better agent are different claims. Plan an improvement with that distinction in mind.