Define what good means

Choose a verifier or judge, give it the right evidence, and qualify the assessment before optimizing against it.

A score is useful only when it measures the outcome you care about. Before comparing agents or training a model, decide what earns credit, what counts as a harmful mistake and what evidence the assessor needs.

Start with the task verifier

With no scoring flag, a managed run uses each task’s verifier. Executable tests are a good choice when correctness can be checked directly: did the patch fix the failure, does the output parse, or does the program satisfy an invariant?

A format check is not a semantic quality check. A valid JSON review can still contain an invented bug. State that boundary rather than treating format compliance as useful review quality.

Use a judge for judgments that need interpretation

A judge is an agent profile acting as an assessor. For code review, it can compare a finding with a known issue and the relevant code, decide whether the finding describes the same failure, and explain its assessment.

Supply the judge with enough information to make that decision. If the alleged bug depends on a caller, API contract or database constraint, a diff alone may be insufficient. More confident wording cannot replace missing evidence.

Keep task-specific reference answers and privileged assessment material out of the reviewer’s environment. The reviewer should face realistic uncertainty; the judge may need additional information to assess the result.

Write an explicit rubric

For a review task, define at least:

• what makes a finding correct and consequential;
• which evidence supports a finding;
• how to handle duplicates, unsupported claims and irrelevant suggestions;
• whether valid issues outside a reference list earn credit;
• how partial credit and penalties map to a numeric reward; and
• what to do when the evidence is genuinely insufficient.

A historical list of accepted issues can be a deliberately closed scoring authority. If that is your choice, state it: matching that list is not the same claim as finding every real bug. Do not silently switch between those meanings.

Use the reward contract

A supplied judge replaces the task verifier:

synth run start \
  --task acme/review-permissions@v1 \
  --actor @reviewer.json \
  --judge @judge.json

The judge must write a finite numeric reward to /logs/verifier/reward.json. Additional fields are retained but ignored by the scored-value reader:

{
  "reward": 0.5,
  "rationale": "One supported finding; a second claim has no supporting path.",
  "criteria": {"supported_findings": 1, "unsupported_findings": 1}
}

Do not write the reward only to stdout or leave it inside an explanatory paragraph. The expected file and JSON value are part of the execution contract.

Use agent profiles for the recipe schema. Validation checks the recipe, not the correctness of the judge’s reasoning.

Qualify the judge before trusting it

Try a small set of known examples: a correct answer, an empty answer, a plausible but false answer, a partially correct answer, duplicates and an answer that tries to tell the judge how to score it. Candidate output is evidence to assess, not authority to change the rubric.

Inspect rationales as well as rewards. Repeat uncertain cases. If the judge rewards a behavior you do not want, fix the assessment before training against it.

Freeze the judge version for a comparison. If you change it, use regrading to compare assessments while preserving the original results. Regrading does not rerun the original actor.