> ## Documentation Index
> Fetch the complete documentation index at: https://synthlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# 1. Get a useful review

> Follow an authorization regression from source to a real agent finding.

export const RecordedReview = () => {
  const [tab, setTab] = useState('Actual review');
  const panels = {
    "Actual review": "{\n  \"findings\": [\n    {\n      \"path\": \"permissions.py\",\n      \"title\": \"Changed AND to OR in can_download, so an admin of one org can download another org's export\",\n      \"explanation\": \"The diff changes can_download from `user.is_admin and user.org_id == export.org_id` to `user.is_admin or user.org_id == export.org_id`. The policy requires both conditions: an export may be downloaded only by an administrator of the organization that owns it, and 'Being an administrator does not authorize access to another organization's export.' With the `or`, the first operand alone short-circuits to True, so any admin of any org passes the check regardless of ownership. For the counterexample, an admin of 'acme' calls download_export on an export owned by 'globex': user.is_admin is True, so `user.is_admin or ...` returns True and the private export is returned, but the policy forbids it because the user is not an admin of the owning org ('globex'). The same root-cause flip also over-permits a non-admin member of the owning org, which the policy likewise requires to be denied.\",\n      \"counterexample\": {\n        \"is_admin\": true,\n        \"user_org_id\": \"acme\",\n        \"export_org_id\": \"globex\"\n      }\n    }\n  ]\n}",
    "Source and caller": "from dataclasses import dataclass\n\n\n@dataclass\nclass User:\n    is_admin: bool\n    org_id: str\n\n\n@dataclass\nclass Export:\n    org_id: str\n    contents: bytes = b\"private export\"\n\n\ndef can_download(user, export):\n    return user.is_admin or user.org_id == export.org_id\n\n\ndef download_export(user, export):\n    if not can_download(user, export):\n        raise PermissionError(\"Export access denied\")\n    return export.contents\n",
    "Policy": "# Export access\n\nAn export may be downloaded only by an administrator of the organization that\nowns it. Both conditions are required. Being an administrator does not authorize\naccess to another organization's export.\n",
    "Work instruction": "# Review the permission change\n\nReview /app/change.diff against the export-access policy in /app/POLICY.md.\nThe current implementation and caller are in /app/permissions.py.\nReport a consequential bug introduced by the change.\n\nWrite /logs/artifacts/review.json with this structure:\n\n```json\n{\n  \"findings\": [\n    {\n      \"path\": \"permissions.py\",\n      \"title\": \"Short description\",\n      \"explanation\": \"Connect the changed code to the failing input and consequence.\",\n      \"counterexample\": {\n        \"is_admin\": true,\n        \"user_org_id\": \"example-org\",\n        \"export_org_id\": \"example-org\"\n      }\n    }\n  ]\n}\n```\n\nThe object above illustrates the format, not a failing input. Your\ncounterexample must be a real input that the changed implementation allows\nbut the stated policy forbids. Use JSON booleans, not strings. Report the root\ncause once; do not duplicate it for several inputs. If you find no supported\nissue, write an empty findings array.\n\nDo not modify the implementation. You may read it and execute small checks.\nThe verifier checks the counterexample against the code and policy; it cannot\nfully assess the quality of your explanation. This is a small teaching example,\nnot an exhaustive test of code-review ability.\n",
    "Score and limits": "{\n  \"reward\": {\n    \"missing_reason\": null,\n    \"present\": true,\n    \"value\": 1\n  },\n  \"assessment\": null,\n  \"scope\": \"Executable counterexample validation only; not independent prose-quality scoring.\",\n  \"meaning\": \"Reward 1 confirms an executable counterexample. No separate prose-quality assessment is present in this run.\"\n}",
    "Provenance": "{\n  \"kind\": \"recorded execution, not simulation\",\n  \"environment\": \"Dev\",\n  \"date\": \"2026-09-26\",\n  \"cli\": \"0.0.1-alpha.62\",\n  \"model\": \"Qwen/Qwen3.8-27B:peft:262144\",\n  \"harness\": \"Claude Code 2.1.263\",\n  \"run_id\": \"3f6be7bd-8fb2-4692-93b9-945ae8d893e5\",\n  \"attempted_trials\": 1,\n  \"scored_trials\": 1\n}"
  };
  return <section className="not-prose rounded-xl border border-zinc-300 dark:border-zinc-700 p-5 my-6" aria-label="Inspect a recorded agent review">
      <p className="mb-3">Recorded execution. Switching views is local: no account, model call or submission.</p>
      <label className="block font-medium">Inspect
        <select aria-label="Inspect recorded evidence" value={tab} onChange={e => setTab(e.target.value)} className="block mt-2 mb-4 rounded border p-2 bg-white text-black">
          {Object.keys(panels).map(name => <option key={name} value={name}>{name}</option>)}
        </select>
      </label>
      <pre tabIndex={0} aria-label={tab} className="whitespace-pre-wrap break-words overflow-x-auto text-sm p-4 rounded bg-zinc-100 dark:bg-zinc-800 max-h-[34rem]">{panels[tab]}</pre>
      <p className="mt-3 text-sm">The score tests the counterexample, not every sentence. Complete readable evidence is linked below.</p>
    </section>;
};

export const PermissionExplorer = () => {
  const [admin, setAdmin] = useState(false);
  const [sameOrg, setSameOrg] = useState(true);
  const before = admin && sameOrg;
  const after = admin || sameOrg;
  return <section className="not-prose rounded-xl border border-zinc-300 dark:border-zinc-700 p-5 my-6" aria-label="Permission counterexample explorer">
      <h3 className="text-lg font-semibold">Try a counterexample</h3>
      <p className="my-2">Local calculation only. No account, model call or saved input.</p>
      <fieldset className="my-4 space-y-3">
        <legend className="font-medium">Who is requesting the export?</legend>
        <label className="flex items-center gap-3"><input type="checkbox" checked={admin} onChange={e => setAdmin(e.target.checked)} />Administrator</label>
        <label className="flex items-center gap-3"><input type="checkbox" checked={sameOrg} onChange={e => setSameOrg(e.target.checked)} />Same organization as the export</label>
      </fieldset>
      <div aria-live="polite" aria-atomic="true" className="rounded-lg bg-zinc-100 dark:bg-zinc-800 p-4">
        <p>Policy and original AND: <strong>{before ? 'Allow' : 'Deny'}</strong></p>
        <p>Changed OR: <strong>{after ? 'Allow' : 'Deny'}</strong></p>
        <p className="mt-2 font-semibold">{before !== after ? 'Regression: the changed code allows a forbidden download.' : 'This input does not demonstrate a regression.'}</p>
      </div>
    </section>;
};

The job is small but consequential: review a change to who can download an organization's export. The policy requires **both** administrator status and the same organization. A caller uses this decision to return the export contents.

```diff theme={null}
 def can_download(user, export):
-    return user.is_admin and user.org_id == export.org_id
+    return user.is_admin or user.org_id == export.org_id
```

The changed condition admits a non-admin in the same organization, and an admin in a different organization. A useful review must name the change, explain the consequence and provide a concrete counterexample—not simply complain about an operator.

<PermissionExplorer />

Text equivalent: neither condition → deny in both versions; both conditions → allow in both; exactly one condition → the policy denies but the changed code allows. Either one-condition case proves the regression.

## A real reference execution

On 26 September 2026, the supplied reviewer produced a finding explaining that an admin of one organization could download another organization's export. Its counterexample had administrator status `true`, user organization `acme`, and export organization `globex`. The retained review also explained the same-organization non-admin path. The executable verifier returned **1**, and the run completed successfully in approximately one minute.

Read the [recorded output and provenance](/downloads/starter-result.json), then explore the finding below. Your review may be worded differently. The [starter project](/downloads/review-permissions.project.json) contains the full source, caller, policy, instruction, profile, environment and verifier. Its score checks the counterexample; it does not independently assess prose quality.

<RecordedReview />

## Run it yourself

After [setting up your CLI](/tutorials/index#before-you-start), [download and unpack the starter project](/tutorials/index#download-and-unpack-a-project) in a new folder. From its new `review-permissions` directory:

```bash theme={null}
python3 check_setup.py
synth task upload ./review-permissions --name my-permission-review --json
```

The setup check checks your Python, CLI, sign-in and agent profile without starting a run. The upload publishes your private task. Save its receipt and copy `version.immutable_ref` into the command below:

```bash theme={null}
synth run start --task <exact-task-ref> --actor @reviewer.json \
  --plan @eval-plan.json --idempotency-key my-permission-review-01 --json
```

This submits one scored trial. The task's executable verifier is used because there is no `--judge`. Keep the returned `run_id`:

```bash theme={null}
synth run show <run-id>
synth run trials <run-id> --wide
```

Wait for a terminal state. Do not submit again while waiting. `succeeded` means the execution finished; it does not mean every review earned a good score.

```bash theme={null}
synth run export <run-id> --output ./first-review-evidence
```

The export verifies and downloads terminal evidence. In `artifacts/`, open the `role-…-output-review` file as JSON, then the assessment in `evaluation.json`. Inspect the finding against `review-permissions/environment/permissions.py`, `review-permissions/environment/POLICY.md` and `review-permissions/environment/change.diff` in your downloaded project.

If you lose a receipt or time out during submission, follow [recovery](/tutorials/operate); do not invent a new idempotency key. An upload or validation success is not execution proof. A missing review or score is a failure to investigate, not a zero-quality review to silently average in.

**What you now have:** a concrete review, its source, a score and a durable record. Next, [find out where that score can mislead](/tutorials/assessment).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.