> ## Documentation Index > Fetch the complete documentation index at: https://synthlabs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Operate and recover safely > Keep run identity through interruptions and know what cleanup actually removes. The downloadable launcher is ordinary Python calling the released `synth` CLI. Inspect `tutorial.py` before running it. There is no browser credential form, hidden paid backend, polling daemon or automatic resubmission. ## Work and spending The course's `python3 tutorial.py ... launch` command is a preview unless you add `--execute`. **Direct `synth run start` and `synth train start` commands submit work immediately** unless you pass `--dry-run`; they do not use the course launcher's preview or workspace ceiling. The workspace's trial limit counts reserved attempts, including uncertain submissions. Preview larger workloads before launching them and keep the returned run IDs. The reference qualification and delivery used **205 reserved trial slots**, including alternative-model qualification, failed attempts, the cancellation canary and the retained-baseline invocation. This is not a required bill for repeating the main path. The core comparison/qualification/training/exam plan uses 136 slots within its explicit 160-slot workspace limit; starter, six calibration controls and adaptations are separate authorized work. Never silently borrow allowance from another workspace. Observed actor medians ranged from 21 to 73 seconds across the matched learning arms; judge medians ranged from 55 to 116 seconds. Allow several minutes for a judged cohort and additional queue/setup time. The recorded two-update training took about 15½ minutes and had 13 missing scores; these observations are not a service-level guarantee. No dollar invoice was available, so the report exposes measured per-role token counts and leaves dollar cost unknown. The core tasks prescribe one CPU, 4 GiB memory, a four-minute actor deadline and a three-minute verifier deadline. Setup, queueing, artifact transfer and cleanup add wall time. These are task-level limits; the course does not override them at run level. A model/harness deadline is not an assurance that a queued run finishes by then. ## Continue without starting again ### Direct CLI starter The first lesson uses the CLI directly. It does not create the course launcher's `state.json` or `operations/` directory. Keep the upload/start receipts, exact task reference, profile, plan and idempotency key together. With the returned run ID, read progress without submitting anything: ```bash theme={null} synth run show synth run trials --wide ``` After it is terminal, export to a new directory with `synth run export --output ./first-review-evidence`. To stop your confirmed run, use `synth run cancel --yes`, then read `show` until it is terminal. Cancellation is not an instantaneous settlement. ### Course launcher ```bash theme={null} python3 -B tutorial.py ./my-course status ``` `state.json`, `operations/` and exported evidence belong to this experiment. Keep them private. Status reads the exact recorded runs. Reloading a web page, closing a terminal or rerunning status does not submit a job. A completed operation reuses its receipt; the launcher refuses changed inputs under the same name and overlapping local writers. ### A submission lost its receipt If a command times out or loses its response, the operation is **uncertain**, not failed. Its reserved work remains counted. Preserve the original arguments and idempotency key (the launcher saves them in `operations/`). `synth run list` and `synth run show` are read-only investigation tools, but a similar name, dataset or timestamp alone does **not** establish that a run belongs to this submission. Have your organization's operator reconcile the original request and exact run identity if the receipt is missing. Do not create a new name/key, rerun the start command, or edit `state.json` to bypass the stop. The launcher has no command to adopt an uncertain submission. Its `status`, `cancel` and `export` operations only address runs whose receipts it recorded. Once the exact run ID is confirmed, use the direct CLI read/cancel/export commands above to settle and preserve that run; keep the uncertain workspace as evidence. The launcher will continue to refuse cleanup until its uncertainty is resolved through supported assistance. This is a safe stop, not an automatic recovery claim. ## Read the phase before fixing anything * No actor output and setup failure: investigate environment/harness setup, not the review prompt. * Actor output but missing score: inspect the verifier/judge and reward contract. * Low score with a completed review: check the actual review and assessment before blaming infrastructure. * Training progress without saved checkpoints: inspect `synth run updates` and `synth run checkpoints`; progress alone is not a usable model. The reference starter encountered both a setup problem and a numeric reward-file packaging error before its successful run. Those attempts remain in the evidence; they are not counted as successful reviews. ## Verify that the evaluation used the checkpoint Save the final checkpoint ID from your training run, then check both its origin and downstream use: ```bash theme={null} synth run checkpoints --json synth checkpoint show --json synth run show --json synth checkpoint runs --operation eval --json synth checkpoint groups --json ``` Require the training checkpoint entry's `checkpoint_id` to equal the chosen ID, and `checkpoint show`'s `checkpoint.source_run_id` to equal your training run. Check the completed update count and saved time. In the evaluation's `show` response, `job.model` must equal that checkpoint ID—not just the base model name. This last check establishes the requested model, not actual consumption by itself. For actual-use readback, the checkpoint's `runs` response must include **your evaluation run ID with `role: "sampled"`**. Its `groups` response must include all expected sampling groups for that run. Follow pagination when `has_more` is true. `synth run groups` and `synth run trials` default to the latest attempt: use `--attempt ` from `synth run show` and join on the full sampling-group ID when checking earlier attempts. `synth run trials --attempt --group --json` reads that exact group. Retries are physical work and may repeat tasks; do not sum them as new logical exam tasks. For each matched group, compare `expected_trial_count`, `settled_trial_count` and `missing_reward_count` with its trial records and your frozen task/sample plan. Missing groups, missing scores, a different ID or an unavailable response mean this check has not passed. Do not substitute an aggregate usage count that might belong to another run. Retain these responses with the export. Checkpoint use proves which model was sampled; only the fixed comparison and inspection of actual outputs can establish improvement. ## Cancel only your named run ```bash theme={null} python3 -B tutorial.py ./my-course cancel practice-baseline --yes python3 -B tutorial.py ./my-course status ``` The first command requests cancellation; the second must show a terminal state before you call it settled. A terminal run is not cancelled again. Export finished evidence before discarding local files. In the recorded early-cancellation exercise, cancellation was requested one second after admission and became terminal about **15 minutes later**. Treat that as an observed recovery delay, not a promise or an instantaneous stop. Keep the exact run ID and continue status checks; never submit a replacement merely because the display still says cancelling. ```bash theme={null} python3 -B tutorial.py ./my-course cleanup ``` Cleanup checks every owned recorded run and refuses an active or uncertain operation. It **does not delete** immutable private task/dataset versions or retained run records: the released CLI has no registry-delete command. It touches neither global configuration nor other runs. Managed execution handles its temporary environments; preserve the experiment's local receipts as the recovery record. Do not advertise this as an account wipe or a spend refund. For an explicit future smoke check, run just the one-trial starter with a new recorded experiment identity and authorized budget. This is the course canary, not a scheduled service. The documentation maintainers own updates to its pinned assets; a failed canary should block a new success claim until repaired and exercised. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.