4. Compare one deliberate change

Freeze tasks and scoring, then inspect what changed in the actual reviews.

The course supplies a baseline profile, reviewer.json, and one deliberate change, reviewer-evidence.json. Both use the same model, harness, tools and task deadline. The changed profile adds a reusable review method: restate the policy, trace the caller, test a concrete counterexample and validate the output before finishing.

This changes the agent’s context, not the tasks, judge or model weights. It is a separate experiment from the model training in the next lesson.

Prepare an isolated workspace

From the review-lab directory after downloading and unpacking the course project:

python3 -B tutorial.py ./my-course prepare --max-trials 160

This checks your sign-in and creates a local workspace with the task inputs, profiles and a recovery ledger. It does not start any runs or change your CLI connection, and it refuses to overwrite an existing directory. The trial limit bounds the total work submitted from this workspace.

The core practice set has eight tasks: buggy/clean pairs covering exact allocation, authorization-cache keys, filtered pagination and signed rounding. The held-out set has eight tasks in four different policy families. Foundational and extension examples are also in the bundle but excluded from the core launch plan. Siblings never cross the split. Full source and policy context are available; labels and assessment stay separate.

Preview, then launch both arms

python3 -B tutorial.py ./my-course launch \
  --name practice-baseline --split practice --trials-per-task 2 --judge
python3 -B tutorial.py ./my-course launch \
  --name practice-evidence --split practice --trials-per-task 2 \
  --actor reviewer-evidence.json --judge

Each preview shows eight distinct tasks and 16 trial slots. Add --execute to the same command only when the workload is authorized. The two arms can run concurrently; there is no need to wait for one to finish before submitting the other. Each successful submission records its own run ID. Do not run two writers against the same local workspace simultaneously.

python3 -B tutorial.py ./my-course status

When both are terminal:

python3 -B tutorial.py ./my-course export practice-baseline
python3 -B tutorial.py ./my-course export practice-evidence
python3 -B report.py ./my-course/evidence/practice-baseline
python3 -B report.py ./my-course/evidence/practice-evidence

Decide from the work, not the headline

Before looking at results, require complete score coverage, more correctly supported bug reviews, no new false findings on clean changes, no semantic regression and no worse timeout rate. With only four practice families and two samples per task, any apparent gain is preliminary—not statistical proof of generalization.

For every task, compare the actual finding and counterexample, the judge’s reasoning, missing results and actor/judge time. The report deliberately says REQUIRES_REVIEW: it cannot automate your judgment by averaging a number. Billing absent from an export is unknown, not free compute.

Repeated trials show variation on a task; they do not create new independent tasks. The reference qualification initially scored perfectly on a simpler set: 16/16 full-credit reviews. We retained that set for instruction and did not use it to claim training gains. The harder practice set was selected before inspecting held-out results.

Read the complete reference experiment, including all attempts and limitations. Do not silently remove failed trials, change the assessor between arms or select held-out tasks because they produce a favorable result.

What the recorded comparison found

Observed on the same eight tasks, two samples each	Original	Evidence-first
Scores present	16/16	16/16
Mean reward	1	1
Buggy-task reviews with supported counterexamples	8/8	8/8
Clean-task reviews with no findings	8/8	8/8
Actor output tokens, all 16 reviews	88,069	67,566
Judge output tokens, all 16 assessments	69,190	60,530
Median actor / judge seconds	48 / 73	32 / 60

Quality improvement: not demonstrated. Keep the original prompt. Every executable counterexample was valid, but reading the explanations exposed errors that the judge missed. One changed rounding review claimed negative half ties round toward negative infinity although its own example demonstrated rounding toward zero; the judge repeated that error. A baseline allocation explanation named the wrong recipient of a lost cent. Full reward did not mean flawless work.

The changed arm used fewer tokens and less observed time. This is not a controlled speed result: median environment setup also changed from 22 to 3 seconds. Retained usage covers all 16 actor and judge sessions per arm; it is not an invoice. Input-token counts include repeated context on successive calls. Dollar cost, optimizer usage and environment billing are unavailable, not zero.

Next, change model weights while keeping the harness fixed.