Observe, budget, and recover: operate the loop when nobody is watching: Practical Field Guide

On this page

01Working definition
02Field situation
03Worked example
04Build it, with checkpoints
05Operating context
06Hands-on lab
07Failure clinic
08Production boundary
09Sources and claim limits

Learning objectives

Define a correlated event model that reconstructs decisions without storing hidden reasoning or sensitive payloads
Enforce step, time, tool, output, change, retry, and spend budgets before dispatch
Detect non-progress and unknown effects from state plus normalized observations
Recover after interruption, tool outage, and approval wait without duplicating work or expanding authority

Before you start

• The verified state machine, tool result union, mechanical rules, and evaluation suite from Modules 04 through 07
• A Release Desk fault injector that can interrupt before and after dispatch, persistence, observation, and checkpoint boundaries

Working definition

Observability and recovery

Operational harnessing makes an agent run reconstructable, bounded, and recoverable. The controller emits correlated events for state transitions, tool dispatch, observations, checks, effects, budgets, and terminal reasons. It reserves budget before work, stops on declared conditions, saves an authoritative checkpoint, and resumes only after identity, policy, environment, Git, state, and effects are reconciled.

A long coding session can appear active while repeating the same search, rerunning an unchanged test, accumulating logs, or waiting on a dead service. Without normalized progress signals and budgets, the model controls both the work and the definition of stuck. The failure reaches a person late, after time and context have already been consumed.

Recovery is also a control boundary. A restart may occur after a write completed but before its observation was saved, after a branch changed, or after an approval expired. Resuming from a conversational summary can repeat an effect or apply stale authority. Checkpoints and effect reconciliation make interruption a tested state transition instead of an improvisation.

Field situation

The six-hour run that made no new progress

The event volume, budget amounts, crash boundaries, and recovery results are synthetic Release Desk fixtures.

Owner: An operator preparing Release Desk coding tasks for background execution with a named human owner.
Decision: How should the controller detect non-progress, stop the first run, and recover the interrupted write without creating a second effect?
Starting state: A session alternates between the same failing browser check and an unchanged route inspection. It produces new prose on each turn, so a text-difference heuristic reports progress. A later crash occurs after audit insertion and before the tool observation is saved.
Expected outcome: The trace ends the repeated loop with a non_progress terminal reason, then a fresh authorized session reconciles the existing audit effect and resumes from the last safe checkpoint.

Constraints

• The controller permits at most 14 model turns, 24 tool calls, 150 seconds, three unchanged failure cycles, one write retry only after reconciliation, and a synthetic USD 2.50 run budget
• Budgets execute outside the model and publish reserved, used, released, and remaining amounts
• Resume must reauthorize the caller and may operate only on the recorded environment and branch
• A kill control must stop new dispatch, cancel owned leases, block resume, and preserve reconciliation evidence

Worked example

Normalized evidence exposed a loop that prose hid

Evidence status: Named synthetic scenario

Three consecutive turns produced different explanations but the same validated action, same target file hash, same browser failure code, same missing evidence, and no state transition. Token and time budgets continued to fall. During a later controlled run, the process crashed after one audit effect committed but before the result reached the agent.

The controller compared normalized action, observation code, evidence coverage, diff hash, and state version. After the third unchanged cycle it stopped with non_progress and a recovery packet. For the crash fixture, resume queried the effect ledger by idempotency key, found the existing audit record, saved the missing observation, and released the reservation without calling the write again.

The receipt showed one logical effect, one physical write, no duplicate retry, and the exact checkpoint used. A kill-control drill then rejected new work, cancelled only the Release Desk lease, blocked resume, and left another fixture environment untouched. Restore required an accountable operator event.

Limits

Thresholds depend on task shape and tool latency. The fixture budgets teach measurement and control placement; a real repository needs observed distributions, service limits, business value, and incident policy.

Method

Build it, with checkpoints

Field situation

How should the controller detect non-progress, stop the first run, and recover the interrupted write without creating a second effect?

01Define the event and budget contracts
02Detect meaningful progress
03Inject failures at persistence boundaries

Acceptance checks

A background run stops when evidence and state stop advancing, every interruption produces a deterministic recovery choice, budgets reconcile, and an operator can inspect or halt work without reading a chat transcript.

Why this visualA timeline across dispatch, effect, observation, checkpoint, crash, reconciliation, and resume makes unknown completion boundaries inspectable.

01
Define the event and budget contracts
Model state, tool, check, effect, budget, checkpoint, approval, and terminal events with correlation and version fields. Define reservation and reconciliation for turns, calls, time, bytes, changed files, retries, and synthetic spend before execution.
CHECKPOINT · A reviewer can reconstruct one fixture run from events, and no seeded canary appears in the model-facing trace or default operator view.
02
Detect meaningful progress
Normalize validated actions, observation codes, evidence coverage, diff hash, failing check, and state version. Stop after the declared unchanged cycles, but reset the counter only when one of those signals advances toward the contract rather than when prose changes.
CHECKPOINT · The repeated-loop fixture terminates at the expected cycle with a recovery packet, while a repair that changes evidence or state continues.
03
Inject failures at persistence boundaries
Crash before dispatch, after dispatch, after effect persistence, after observation persistence, and after checkpoint. Restart with a fresh session and validate identity, policy, environment, branch, contract, state, budget reservation, lease, and effect ledger before choosing retry, reconcile, resume, or stop.
CHECKPOINT · All five fixtures reach the expected terminal state with one logical effect, correct reservation accounting, and no unowned environment change.
04
Exercise kill, restore, and operator handoff
Trigger the global fixture kill control during an active run. Verify intake, dispatch, leases, and resume behavior, preserve in-flight evidence, then perform a named restore and continue from reconciliation. Give the packet to a reviewer who did not watch the run.
CHECKPOINT · The reviewer identifies the stop reason and safe next action, the target run remains bounded, and the unrelated environment continues normally.

Operating context

Beyond the demo

Record events that answer operating questions

Start from questions a maintainer must answer: which contract and commit ran, what state changed, which tool executed, what evidence returned, where budget went, why the run stopped, which effects may exist, and what action is safe next. Give every event run, session, task, environment, contract, commit, feature, span, and parent identifiers as applicable.

Store validated arguments or hashes and safe projections, not credentials, raw session cookies, unrestricted source files, or hidden model reasoning. Link protected logs and screenshots by evidence ID with retention and access policy. A useful trace reconstructs control flow and facts while minimizing sensitive context.

Make recovery a fixture suite

Inject interruption before dispatch, after dispatch but before receipt, after effect persistence, after observation persistence, and after checkpoint. For each boundary, declare expected state version, logical effect count, lease, budget reservation, next action, and terminal reason. Unknown writes reconcile; safe reads may use a bounded retry policy.

A resume begins like a new session: validate caller authority, environment ownership, branch and commit, active contract, checkpoint version, outstanding lease, pending approval, budget, and effect ledger. Any mismatch routes to a named blocked or reconcile state. Resume does not inherit authority simply because an earlier session had it.

Hands-on lab

Instrument and crash the Release Desk run

Add an event schema, budget ledger, non-progress detector, checkpoint validator, effect reconciliation, and kill control. Execute five crash boundaries plus one repeated-loop fixture.

Prepare

• Seed a fixture-only canary in input, tool output, environment, and screenshot metadata so trace redaction can be tested
• Commit a healthy checkpoint and record the expected state, branch, environment, effect count, and budgets before fault injection

Deliverable

A versioned event contract, trace viewer fixture, multi-dimensional budget ledger, progress detector, five crash receipts, reconciliation test, redaction report, kill and restore drill, and recovery runbook.

Starter kit: Harness event envelope

JSON Schema

{"type":"object","required":["eventId","runId","taskId","type","occurredAt","payloadVersion"],"properties":{"type":{"enum":["state.changed","tool.dispatched","tool.observed","check.completed","effect.reconciled","budget.changed","run.terminated"]},"evidenceIds":{"type":"array","items":{"type":"string"}}}}

Downloadable artifacts

Harness event schema

harness-event.schema.json · JSON Schema

An editable course fixture for the main lab. Save it inside the Release Desk repository before running the acceptance command.

Run receipt template

he-08-receipt.json · JSON

A compact evidence record for the check, environment, result, and limits that another reviewer must be able to inspect.

Acceptance command

npm run harness:recovery -- --faults fixtures/crash-boundaries.jsonl --kill-drill

Expected receipt

PASS he-08 observability-recovery
crashBoundaries=5 logicalEffects=1 duplicateWrites=0
canaryHits=0 killScopeViolations=0 budgetsReconciled=true

Expected result

Carry forward

Use event and budget distributions to decide whether parallel execution is justified. Carry the recovery runbook, kill drill, and terminal reason taxonomy into the capstone.

Acceptance checks

01Events reconstruct contract, commit, environment, state, tools, checks, effects, budgets, terminal reason, and next action
02The non-progress fixture stops after the declared unchanged cycles and cannot be fooled by paraphrased model text
03Five crash boundaries preserve one logical effect and choose retry or reconciliation according to effect certainty
04Kill and restore affect only owned resources, block stale resume authority, preserve evidence, and produce named operator events

What breaks

Failure clinic

F1The trace is large but cannot explain why the run stopped.

Inspect: Check correlation IDs, state versions, terminal events, budget changes, effect records, evidence links, and schema consistency.
Likely cause: The system logged raw text volume without recording control decisions and authoritative state transitions.
Repair: Add typed events for the missing decision boundary and link existing safe diagnostics through evidence IDs.
Prevent next time: Test trace reconstruction as an acceptance case and remove fields that do not answer an operating question.

F2A resumed session repeats a write that completed before the crash.

Inspect: Compare dispatch reservation, idempotency key, effect ledger, checkpoint, observation persistence, and retry policy.
Likely cause: Resume trusted missing observation as proof that the effect did not happen.
Repair: Reconcile against the authoritative effect record, persist the recovered observation, and prevent physical re-dispatch.
Prevent next time: Inject crashes on both sides of every consequential effect boundary and assert logical plus physical counts.

F3The kill switch stops an unrelated local project.

Inspect: Review environment identity, lease ownership, resolved process metadata, ports, paths, and the exact kill target list.
Likely cause: The control used a broad port or process match instead of task-scoped ownership records.
Repair: Restore the unrelated process, narrow the resolver to explicit environment ownership, and add a dry-run collision fixture.
Prevent next time: Print and validate exact targets before action; never terminate from a broad path, unresolved variable, or port alone.

Beyond the demo

Production boundary

01Event schemas capture control and evidence with stable correlation, versioning, redaction, retention, and access policy
02Turn, call, time, byte, change, retry, and spend budgets reserve before dispatch and reconcile after outcome
03Progress uses normalized action, observation, evidence, diff, and state signals rather than textual novelty
04Every terminal reason maps to an operator packet, safe next action, owner, and allowed resume behavior
05Recovery revalidates identity, permission, environment, Git, contract, state, lease, budget, approval, and effects
06Kill, restore, crash, outage, unknown effect, stale authority, and unrelated-environment collision fixtures run regularly

Evidence status

Sources and claim limits

Sources support the named claims; they do not guarantee the same result in another system.

[1]
Effective harnesses for long-running agents
Anthropic · Published research · 2026-08-26
initializer pattern · feature ledger · session handoff · end-to-end verification
[2]
Harness design for long-running application development
Anthropic · Published research · 2026-08-26
planner-generator-evaluator · testable contracts · harness simplification · cost tradeoffs
[3]
Harness engineering: leveraging Codex in an agent-first world
OpenAI · Public case · 2026-08-26
repository knowledge · agent legibility · mechanical enforcement · entropy management
[4]
Harness Engineering Guide
Nexu · Public case · 2026-08-26
runtime boundary · tool systems · sandboxing · recovery patterns
[5]
Building effective agents
Anthropic · Official documentation · 2026-08-26
simplest viable architecture · workflow patterns · environmental feedback · stop conditions