1. Get a useful review

Follow an authorization regression from source to a real agent finding.

The job is small but consequential: review a change to who can download an organization’s export. The policy requires both administrator status and the same organization. A caller uses this decision to return the export contents.

 def can_download(user, export):
-    return user.is_admin and user.org_id == export.org_id
+    return user.is_admin or user.org_id == export.org_id

The changed condition admits a non-admin in the same organization, and an admin in a different organization. A useful review must name the change, explain the consequence and provide a concrete counterexample—not simply complain about an operator.

Text equivalent: neither condition → deny in both versions; both conditions → allow in both; exactly one condition → the policy denies but the changed code allows. Either one-condition case proves the regression.

A real reference execution

On 26 September 2026, the supplied reviewer produced a finding explaining that an admin of one organization could download another organization’s export. Its counterexample had administrator status true, user organization acme, and export organization globex. The retained review also explained the same-organization non-admin path. The executable verifier returned 1, and the run completed successfully in approximately one minute.

Read the recorded output and provenance, then explore the finding below. Your review may be worded differently. The starter project contains the full source, caller, policy, instruction, profile, environment and verifier. Its score checks the counterexample; it does not independently assess prose quality.

Run it yourself

After setting up your CLI, download and unpack the starter project in a new folder. From its new review-permissions directory:

python3 check_setup.py
synth task upload ./review-permissions --name my-permission-review --json

The setup check checks your Python, CLI, sign-in and agent profile without starting a run. The upload publishes your private task. Save its receipt and copy version.immutable_ref into the command below:

synth run start --task <exact-task-ref> --actor @reviewer.json \
  --plan @eval-plan.json --idempotency-key my-permission-review-01 --json

This submits one scored trial. The task’s executable verifier is used because there is no --judge. Keep the returned run_id:

synth run show <run-id>
synth run trials <run-id> --wide

Wait for a terminal state. Do not submit again while waiting. succeeded means the execution finished; it does not mean every review earned a good score.

synth run export <run-id> --output ./first-review-evidence

The export verifies and downloads terminal evidence. In artifacts/, open the role-…-output-review file as JSON, then the assessment in evaluation.json. Inspect the finding against review-permissions/environment/permissions.py, review-permissions/environment/POLICY.md and review-permissions/environment/change.diff in your downloaded project.

If you lose a receipt or time out during submission, follow recovery; do not invent a new idempotency key. An upload or validation success is not execution proof. A missing review or score is a failure to investigate, not a zero-quality review to silently average in.

What you now have: a concrete review, its source, a score and a durable record. Next, find out where that score can mislead.