Build a reviewer you can evaluate

Run useful work, challenge its assessment, make one change, and decide whether training helped.

Imagine you are building a code reviewer for DispatchDesk, a fictional application with permissions, billing and queue policies. You want it to catch policy regressions without inventing bugs.

In this course, you will run the reviewer, inspect its findings, define what counts as correct, and compare changes on work it did not train on. The complete teaching codebase is supplied under the MIT license.

Code review is the example throughout. For a support agent, the task might be a customer case and a resolution policy; for a document analyst, a document and its source records; for an engineering assistant, project documents and requirements. The task, agent profile and assessment change, while the workflow stays the same.

You bring	Synth does	You get
Work instructions, source and an output contract	Runs the chosen agent in each task environment	Review files, execution records and failures
A qualified verifier or judge and a task collection	Repeats the work and records assessments	Evidence for comparing agents
Practice data, a trainable agent and an explicit plan	Optimizes the model and saves checkpoints	A model to evaluate—not a guaranteed improvement

Agent = Model + Harness + Context. The task environment is separate. A profile pins the agent; a task supplies work and an environment; a dataset selects task versions; a run records what happened. You remain responsible for relevant tasks, lawful data use, meaningful assessment and the promotion decision.

Choose how to begin

Explore the recorded work

See the source, a real review and its assessment.

Run the course

Download the complete inputs and use the released CLI in a fresh workspace.

Before you start

Create a Synth account, install the CLI and sign in.

You will also need Python 3.10 or later and a macOS or Linux terminal for the example scripts. The downloads include the tasks, agent profiles and assessment tools.

Start with one trial. Later lessons show how to preview larger experiments and keep their inputs and results together.

Six connected lessons

1. Get a useful review: read an actual finding and reproduce it.
2. Challenge the score: expose a verifier blind spot and calibrate a judge.
3. Bring your own failure: build a complete, inspectable task.
4. Compare one deliberate change: freeze inputs and inspect paired evidence.
5. Train and test transfer: distinguish an update, a checkpoint and improvement.
6. Use or reject the result: apply the decision and retain a rollback path.

Every lesson uses the same course project and manifest. You can start later with the supplied files; recorded run IDs are provenance, not prerequisites or resources in your account. Operation and recovery covers interruptions and cleanup.

Download and unpack a project

The versioned example release contains ordinary ZIP downloads and SHA-256 checksums. Inspect the complete source before executing it. Download the starter into a new directory:

mkdir synth-starter
cd synth-starter
curl --fail --location --remote-name https://github.com/SynthLabsAI/synth-examples/releases/download/v2026.09.27.2/review-permissions.zip
curl --fail --location --remote-name https://github.com/SynthLabsAI/synth-examples/releases/download/v2026.09.27.2/checksums.txt
shasum --algorithm 256 --check --ignore-missing checksums.txt
unzip review-permissions.zip -d review-permissions
cd review-permissions

The checksum check must say review-permissions.zip: OK. For the complete course, use review-lab.zip and extract into a new review-lab directory instead. Downloading and extracting does not authenticate or submit work. Keep the release tag and checksum file with your inputs; do not extract over an existing project.

Alternative: inspect or unpack the readable JSON project

Download either the starter project or the complete course into a new folder, preserving its .project.json filename. These are readable JSON files containing the complete source and a checksum for every file—not executable installers or encoded archives.

In that folder, run the command below. It validates all file paths and checksums, refuses to replace an existing project, and writes the supplied files. It does not authenticate, submit work or run downloaded code.

python3 - <<'PY'
"""Unpack a public JSON project into a new directory; never execute its files."""
import hashlib
import json
import shutil
from pathlib import Path, PurePosixPath


def unpack(source, parent):
    if source.stat().st_size > 20 * 1024 * 1024:
        raise ValueError('Project exceeds the 20 MiB input limit')
    project = json.loads(source.read_text(encoding='utf-8'))
    if project.get('schema') != 'synth.tutorial-project.v1':
        raise ValueError('Unsupported project format')
    if project.get('name') not in {'review-lab', 'review-permissions'}:
        raise ValueError('Unexpected project name')
    files = project['files']
    if not isinstance(files, dict) or not files or len(files) > 2000:
        raise ValueError('Invalid project file inventory')
    checked = []
    for name, record in files.items():
        path = PurePosixPath(name)
        if path.is_absolute() or '..' in path.parts or '\\' in name or str(path) != name or name == '.' or any(ord(c) < 32 for c in name):
            raise ValueError('Unsafe project path: ' + name)
        data = record['text'].encode('utf-8')
        if hashlib.sha256(data).hexdigest() != record['sha256']:
            raise ValueError('File checksum mismatch: ' + name)
        checked.append((path, data))
    paths = {path for path, _ in checked}
    if any(parent in paths for path in paths for parent in path.parents):
        raise ValueError('Conflicting file and directory paths')
    destination = parent / project['name']
    destination.mkdir(exist_ok=False)
    try:
        for path, data in checked:
            target = destination / str(path)
            target.parent.mkdir(parents=True, exist_ok=True)
            with target.open('xb') as stream:
                stream.write(data)
    except BaseException:
        shutil.rmtree(destination)
        raise
    return destination


if __name__ == '__main__':
    projects = list(Path('.').glob('*.project.json'))
    if len(projects) != 1:
        raise SystemExit('Use a new folder containing exactly one downloaded .project.json file')
    print('Project ready:', unpack(projects[0], Path('.')))
PY

For the starter, continue from the new review-permissions directory. For the full course, continue from review-lab. Keep the downloaded JSON and shared manifest as the record of your inputs.