--- title: Durable runs summary: A journaled run resumes where it stopped. runId is the idempotency key. canonical: https://docs.caveman.so/docs/agent-sdk/durable license: Apache-2.0 capability: agent-sdk updated: 2026-09-16T21:32:40-07:00 basis: inferred --- # Durable runs > A journaled run resumes where it stopped. runId is the idempotency key. A durable run writes an append-only journal before it does any network work, so the run resumes from where it stopped and the same `runId` submitted twice buys its tokens once. Turn it on with one option on `run()`. ```ts import { DiskDurableStore, agent, auto, run } from "@caveman-ai/agent"; const support = agent({ id: "support", instructions: "Answer from policy.", model: auto() }); const store = new DiskDurableStore("./.caveman/runs/durable"); const first = await run(support, "Where is order A-123?", { durable: { runId: "nightly-2026-09-16", store }, }); const again = await run(support, "Where is order A-123?", { durable: { runId: "nightly-2026-09-16", store }, }); console.log("first.text ", JSON.stringify(first.text)); console.log("second.text ", JSON.stringify(again.text)); console.log("second.receipt ", again.receipt.calls.length, "calls"); console.log(); console.log("--- journal event types"); for (const line of await store.load("nightly-2026-09-16")) { const event = JSON.parse(line); console.log(event.type, Object.keys(event).join(",")); } ``` Unreleased: this page documents `@caveman-ai/agent` 0.2.0 from source at commit 311be40 in caveman-ai/agent-sdk. Build it from the checkout as described on [Install](/docs/agent-sdk/install). Captured from that script against a scripted model from `@caveman-ai/agent/testing`: ```console first.text "Order A-123 shipped on Tuesday." second.text "Order A-123 shipped on Tuesday." second.receipt 1 calls --- journal event types run_started v,at,type,runId,agentId,definitionSha256,input,sessionId,denomination,budgetSha256,pid call_started v,at,type,path,kind,provider,model call_settled v,at,type,path,kind,call turn v,at,type,messages run_completed v,at,type,result ``` One `call_started` for two `run()` calls: the second one read the journaled result and returned it. The store writes `0o700` directories and `0o600` files, because a journal holds the conversation it needs to resume, unlike a receipt, which is content-blind. ## runId is the idempotency key The caller assigns it, and it must match `[A-Za-z0-9][A-Za-z0-9._-]{0,127}`. The same id always means the same logical run. | Journal state | What `run()` does | |---|---| | No journal | Starts fresh and writes `run_started`. | | No terminal event | Resumes from the last turn checkpoint, preloading spend already journaled. | | `run_completed` | Returns the journaled result. No provider call. | | `run_failed` | Throws the journaled error again, with its partial receipt. | Identity is checked before a resume spends anything, and a mismatch throws instead of quietly becoming a different run: `cave_durable_definition_changed` when the definition digest moved, `cave_durable_input_mismatch` for different input under the same id, `cave_durable_budget_changed` when the money contract changed, `cave_durable_agent_mismatch`, `cave_durable_session_mismatch`, `cave_durable_conversation_mismatch`. A new task needs a new id. ## Resuming after a crash Kill the process during a provider call and re-run the same id: ```console killed the process mid-call status: pending attempts: 1 input replayable: true text: "Order A-123 shipped on Tuesday." receipt.resume: { "attempts": 2, "priorCalls": 0, "priorEstimatedUsd": 0, "priorTokens": 0, "priorUnpriced": false, "possibleDoubleCountCalls": 1, "discardedPartialTurn": false } ``` `attempts: 2` counts this attempt. `priorCalls` and `priorEstimatedUsd` are what earlier attempts settled, restored into this run's meter so a crash leaves the budget where those attempts left it. `priorSettled` appears when the run carries a budget. Conversation state checkpoints at turn boundaries. A partial turn is discarded and re-driven, and `discardedPartialTurn` says so; its journaled spend is kept either way. Tools resume by effect. A `read` tool may re-drive. An `idempotent` tool re-drives with a stable key derived from the run id, the tool path, the call id, and the SHA-256 of its arguments. A `write` or `external` tool whose intent was journaled and whose settlement was not stops the resume with `cave_durable_tool_effect_uncertain::`, because the journal records the intent and stops short of the outcome. Declaring effects honestly is what makes this work: [Tools](/docs/agent-sdk/tools). **possibleDoubleCountCalls is the at-least-once ceiling** A provider call journals its intent before the request leaves and its usage after the response returns. A crash in between leaves one call the provider may have billed and this ledger never saw. That call is counted in `possibleDoubleCountCalls` and in no other figure on the receipt, so `priorCalls`, `priorEstimatedUsd` and the total all exclude it. The guarantee is at-least-once at the step boundary, and the receipt says which calls are uncertain rather than picking a number. ## Reading a journal without resuming it `durableRunSummary(lines)` classifies a journal. It reads and returns, which is why a status endpoint or a recovery sweep can use it freely. `support` here is the definition from the first block: ```ts import { DiskDurableStore, durableInputIsReplayable, durableRunSummary, run } from "@caveman-ai/agent"; const store = new DiskDurableStore("./.caveman/runs/durable"); const summary = durableRunSummary(await store.load("nightly-2026-09-16")); if (summary.status === "pending" && durableInputIsReplayable(summary.input)) { await run(support, summary.input, { durable: { runId: summary.runId, store } }); } ``` `status` is `missing`, `pending`, `completed`, or `failed`. A pending summary carries `runId`, `agentId`, `input`, `startedAt`, `attempts`, and optionally `cancelRequested` and `wakeAt` with `sleepReason`. A completed one carries `result`, a failed one `code`, `message`, and `receipt`. The search covers the whole journal rather than the tail, so a settled run reads as settled whatever was appended after it. `durableInputIsReplayable(input)` is the guard before an unattended resume. A multimodal run journals the digest of its lowered context rather than the content, so rebuilding that input takes the original from the caller. [Deploy](/docs/agent-sdk/deploy) reports that case as `cave_serve_resume_needs_original_input`. ## The stores Four stores ship, all from `@caveman-ai/agent/durable`. Every store is the same four methods: `load`, `append`, `acquire`, `close`, plus an optional `list()` that crash recovery needs. `append` resolving means the bytes survive a crash. ### Disk ```ts new DiskDurableStore("./.caveman/runs/durable"); ``` The default when `durable.store` is omitted, rooted at `/.caveman/runs/durable`. One directory per run, suffixed with a digest so two ids differing only in case stay separate journals on macOS and Windows. ### HTTP ```ts import { HttpDurableStore } from "@caveman-ai/agent/durable"; new HttpDurableStore({ url: process.env.CAVE_JOURNAL_URL!, token: process.env.CAVE_JOURNAL_TOKEN! }); ``` The journal lives in a service you run. The URL must be `https` outside `localhost` and `127.0.0.1`, or the constructor throws `cave_durable_http_insecure: journal URL must be https outside localhost`. A missing token throws `cave_durable_http_token_required`. The lock is a lease, 30 seconds by default, renewed at a third of its length; `lockTtlMs` and `requestTimeoutMs` (10 seconds) are both configurable. ### SQL ```ts import { SqlDurableStore } from "@caveman-ai/agent/durable"; const store = new SqlDurableStore({ sql: { exec: (query, params) => db.prepare(query).all(...params) }, dialect: "sqlite", }); ``` The whole database dependency is `exec(sql, params)`, sync or async, returning rows, which is the shape every driver already has. `dialect` picks the placeholder grammar: `?` for sqlite, `$1…$n` for postgres. `table` defaults to `caveman_durable_journal` and the lease table is `_leases`. Run the DDL once yourself: ```ts await db.exec(SqlDurableStore.schema("sqlite")); ``` DDL stays with you, because a journal store that can create tables can also drop them. ### Object storage ```ts import { ObjectDurableStore } from "@caveman-ai/agent/durable"; const store = new ObjectDurableStore({ storage: { get: (key) => bucket.get(key), put: (key, data, opts) => bucket.put(key, data, opts), list: (prefix) => bucket.list(prefix), }, conditionalPut: true, }); ``` The constructor takes one `ObjectStorage`, three methods over a bucket: `get(key)`, `put(key, data, opts)`, `list(prefix)`. S3, R2 and GCS all have them. Keys land under `prefix`, which defaults to `caveman/durable/`, one journal chunk and one lease object per run. `conditionalPut: true` declares that your `put` treats `ifMatch: ""` as create-if-absent, which is how the lease is claimed. `acquire()` throws `cave_durable_object_conditional_put_required` without it. `leaseTtlMs` defaults to 30 seconds and has a floor of 3,000 ms, below which the constructor throws `cave_durable_object_lease_ttl_invalid: lease must be at least 3000ms`. ## Limits One process drives a run at a time. `acquire(runId)` takes an exclusive lock and throws `cave_durable_run_locked` while another live process holds it. The HTTP and SQL stores hold that lock as an expiring lease, so a process that cannot renew stops being able to append rather than writing into a journal someone else now owns. Durability is a root-run contract. Subagents share the root journal rather than opening their own, so money from every depth lands in one ledger. `durable` cannot be combined with `maxCostUsd`; use `budget` instead. Breaker windows restart on resume. Money figures in a journal are revalidated on the way out with the same checks the live meter uses, because a store is a trust boundary rather than only a local file. Every figure stays a public-catalog list-price subtotal labelled `inferred`: [Numbers and limits](/docs/counting).