Measure results
Evaluate success, latency, cache effects, retries, and usage.
Evaluate middleware on completed tasks, not the number of applied reports. A shorter tool result can increase recovery calls, retries, latency, or output tokens. Decision reports do not establish savings.
Reproduce the mechanism first#
Run the two downloadable quickstarts with DEMO_MODE=off, record, and compress. Use the same deterministic log and scripted model. Compression mode must show a replacement, execute the native recovery tool, reconstruct the original exactly, and retain application history. No provider calls occur in these default runs.
This proves local mechanism behavior with a real Engine and framework. It does not test a provider's acceptance, answer quality, cache accounting, or billed cost. The scripted AI SDK model's zero token counts are fixture data, not observed provider usage.
Freeze an evaluation set#
Choose a fixed set of real tasks with explicit pass/fail criteria. Keep model IDs, provider versions, tool implementations, prompts, maximum steps, timeouts, and retry policy fixed. Save original tool outputs so each mode sees the same inputs. Do not reuse one mutable provider conversation across modes.
Run off first as the native baseline. Use record to see eligibility without text changes, then compress. Randomize run order or interleave modes to reduce time-of-day effects. Repeat each task enough times to expose nondeterministic model behavior. Count failed and timed-out attempts in the results.
Evaluate cold and warm cache cases separately. For cold runs, use fresh provider conversation/cache state where supported and a new Caveman scope. For warm runs, deliberately replay the same append-only conversation and stable scope. Record provider cache usage when returned; stable Caveman choices alone do not prove a provider cache hit. Do not modify prompts merely to create a hit while comparing different tasks.
Record one row per attempt#
task_id,mode,cache_condition,attempt,success,elapsed_ms,provider_requests,retries,recovery_calls,input_tokens,output_tokens,cache_read_tokens,cache_write_tokens,usage_complete,decision_status,decision_reasonUse a monotonic timer around the entire native loop, including optimization, tools, recovery, provider retries, and final response. Separately time individual model and tool calls when diagnosing overhead. Count automatic provider/framework retries, not only explicit application retries. Include recovery tool outputs in subsequent provider input usage.
| Metric | Source | What it can prove |
|---|---|---|
| Final decision | onReport / on_report, or latest local report | Whether the adapter used replacements |
| Segment token estimate | Explicit runtime.optimize() result's plan.measurement | Inferred local segment counts, when a plan exists |
| Provider usage | Native provider response, framework-normalized usage, or final stream usage | Observed counts returned for that request, subject to completeness |
| Cache use | Provider cache read/write usage fields | Reported cache effects for that request |
| Recovery calls and retries | Your loop/tool hooks and provider attempt instrumentation | Extra work induced by the run |
| Latency | Monotonic application timer | End-to-end elapsed time |
| Task success | Your independent evaluator | Whether the task met its acceptance criteria |
| Billing | Provider invoice/ledger for matching traffic | Billed amounts; local middleware does not verify them |
Native wrappers do not expose every internal plan through CallReport. Do not run a second optimizer request and present its estimate as the plan used by the first native call. If your chosen boundary does not expose a metric, record it as unavailable. Never substitute zero for missing usage.
For AI SDK, retain per-step usage and totalUsage; for LangChain, inspect AIMessage.usage_metadata when the selected provider supplies it. A prematurely closed stream may lack final usage. Application retries can generate charges even when your final response is a failure. Connected tracing is optional and requires its own endpoint; it is not necessary for local decision logging.
Compare results honestly#
Report pass rate, latency distribution, provider requests, retries, recovery rate, and complete usage by mode and cache condition. Compare costs only when provider pricing and usage provenance support that calculation, and label such calculations estimates. Cost per successful task must include costs of failed attempts and retries.
Do not multiply one run into a monthly claim. inferred is a local estimate, measured is observed traffic, and verified is reserved for systems that can compare against a bill. Nothing in these local examples emits verified savings. See counting and explicit compression for the shared vocabulary.