Resilience

Production traffic peaks Monday morning. Anthropic returns 429s for the next 90 seconds. Your support agent has 200 concurrent users and zero patience for a backoff loop. The framework's resilience decorators wrap any provider with retry + fallback so your agent degrades gracefully instead of throwing user-visible errors.

Four composable decorators

All resilience decorators are exported from the agentfootprint/resilience subpath. Provider factories (anthropic, openai, bedrock, …) live at agentfootprint/providers.

All four preserve the LLMProvider interface — drop-in replacements for the underlying provider. They compose freely:

import { withRetry, withFallback } from 'agentfootprint/resilience';
import { anthropic, openai } from 'agentfootprint/providers';

const provider = withRetry(
  withFallback(
    anthropic({ apiKey: process.env.ANTHROPIC_API_KEY! }),
    openai({ apiKey: process.env.OPENAI_API_KEY! }),
  ),
  { maxAttempts: 5 },
);

Reads as: try anthropic; on failure fall back to openai; the whole chain is wrapped in retry with 5 attempts. Right-fold of withFallback + outer withRetry is the standard production composition.

Convenience: fallbackProvider

For the common N-providers-in-a-chain shape, use the variadic factory. It is sugar over repeated withFallback — tries each provider in order, advancing on errors that match the (optional) shouldFallback predicate; the first success wins, and if all fail the last error throws. Wrap the whole chain in withRetry for retries:

import { fallbackProvider, withRetry } from 'agentfootprint/resilience';
import { anthropic, openai, bedrock } from 'agentfootprint/providers';

const provider = withRetry(
  fallbackProvider(
    anthropic({ apiKey: process.env.ANTHROPIC_API_KEY! }),
    openai({ apiKey: process.env.OPENAI_API_KEY! }),
    bedrock({ region: 'us-west-2' }),
  ),
  { maxAttempts: 3, initialDelayMs: 200, backoffFactor: 2 },
);

Pass an options object as the FIRST argument to customize the chain (shared shouldFallback/onFallback, or an explicit name):

const provider = fallbackProvider(
  { name: 'llm-chain', onFallback: (err) => console.warn('falling back', err) },
  anthropic({ apiKey: process.env.ANTHROPIC_API_KEY! }),
  openai({ apiKey: process.env.OPENAI_API_KEY! }),
);

The default retry policy (and overriding it)

By default withRetry skips AbortError and HTTP 4xx client errors (those don't benefit from retry) — except 429 Too Many Requests, which IS retried — and retries everything else (5xx, network errors, unknown shapes) while attempt < maxAttempts. It reads err.status (or err.statusCode) when present, and retries when the status is unknown. Override shouldRetry to implement a custom policy:

import { withRetry } from 'agentfootprint/resilience';

withRetry(provider, {
  maxAttempts: 5,
  shouldRetry: (err, attempt) => {
    const status = (err as { status?: number }).status;
    if (status === 429) return true;            // always retry rate limits
    if (status && status >= 500) return attempt < 3; // cap server errors at 3
    return false;
  },
});

shouldFallback(err) works the same way for withFallback — its default falls back on every error except AbortError.

Observability: automatic inside a run

Which provider actually served this call? — the first question anyone asks after a vendor blip. Inside an Agent, LLMCall, or Parallel run the answer is in the trace, with no wiring. Each decorator reports what it did through an optional per-call LLMCallHooks, and the LLM call site turns that report into a typed event from inside the traversal — so it carries the real runId and runtimeStageId footprintjs stamped before the stage ran, and correlates with every other event in the run.

Subscribe BEFORE run(). The dispatcher drops events nobody is listening for, so a late listener looks exactly like a missing emitter:

const provider = withFallback(  downProvider('acme-llm', 'acme 503: gateway timeout'),  healthyProvider('backup-llm', 'Everything is up.'),);const agent = Agent.create({ provider, model: 'mock', maxIterations: 2 }).build();// Subscribe BEFORE run() — the dispatcher DROPS events nobody listens for,// so a late listener looks exactly like a missing emitter.const failovers: Failover[] = [];agent.on('agentfootprint.fallback.triggered', (e) => {  failovers.push({    primary: e.payload.primary, // the vendor that failed    fallback: e.payload.fallback, // the vendor that actually served    reason: e.payload.reason, // the failing vendor's error message    stage: e.meta.runtimeStageId, // real, stamped by footprintjs    runId: e.meta.runId,  });});const reply = await agent.run({ message: input });

agent.on('agentfootprint.error.*', …) takes the retry family in one subscription; '*' takes everything.

Which recorder surfaces them

resilienceRecorder() — a small event bridge matched on those three exact event names. Agent, LLMCall and Parallel are the runners that call a provider themselves, and each attaches it for you, so .on() works with zero setup. Two paths need a hand:

• A bare FlowChartExecutor running one of the exported message-api charts (buildAgentMessageApiChart / buildMessageApiChart). The chart calls $emit, but with no recorder attached that call reaches nothing at all — footprintjs's emit channel dispatches only to recorders' onEmit, so the report never lands in the commit log, the snapshot, or the run's return value. Attach a bridge yourself: executor.attachCombinedRecorder(resilienceRecorder({ dispatcher, getRunContext })), exported from agentfootprint/observe.
• A saved run. recordRun() captures these like any other event, meta intact, so an offline viewer can place the failover on the same timeline as the tool calls around it. See Offline replay.

One producer per fact, so nothing double-counts

Each event has exactly one producer — fallback.triggered ← withFallback, error.retried / error.recovered ← withRetry, error.circuit_changed ← withCircuitBreaker. Decorators only forward the hooks inward, never aggregate or re-emit, so a stack of three produces one concatenated stream and de-duplication is structurally unnecessary. A production stack (breaker under fallback under retry) whose primary is down and whose backup blips once reads:

error.circuit_changed → fallback.triggered → error.retried → fallback.triggered → error.recovered

Two fallbacks because there really were two billed primary calls — not a duplicate. fallbackProvider(a, b, c) reports honest pairwise hops (a → b|c, then b → c), never the composite chain name.

The honest limits

• error.retried.reason classifies the ERROR, not the decision. shouldRetry returns a bare boolean, so a custom predicate's reasoning is unknowable to the decorator. The value is derived from the same status/statusCode fields the default policy inspects: 'http-429', 'http-5xx', 'http-4xx', `http-${code}`, or 'no-status'.
• error.circuit_changed reports TRANSITIONS, not calls. An open breaker rejecting a hundred requests produces zero events — a re-entry into the same state is not a change, and an event per rejection would turn a state log into a request log. Its reason is the breaker's own words ('3 consecutive failures', 'cooldown elapsed'), never the failing vendor's message: the error that tripped it is reported by whoever threw it.
• onStateChange and the event are complements, not duplicates. The hook fires wherever the breaker lives, in a run or not, which is why a Redis-backed counter belongs on it. The event fires only inside a run, which is why it can carry real correlation ids. Outside a run nothing passes hooks and the breaker reports nothing at all — standalone behaviour is byte-identical to before 9.32.
• A fallback success is reported as fallback.triggered, not error.recovered. That payload has no field for which provider served, so claiming a recovery there would be inventing data. withRetry is the only source of recovered, and only for an attempt ≥ 2.
• error.recovered.totalDurationMs is new instrumentation, measured by the decorator from its own first attempt — not a fact recovered from somewhere else.
• Streaming reports only a pre-first-chunk failure. Once the primary's stream has yielded a chunk the stream is pinned; a later error re-throws rather than failing over, so there is nothing to report. A primary with no stream() at all reports nothing either — nothing failed.
• The memory extractors are a blind spot. The LLM-backed beat / fact extractors call complete() from inside a port with no scope, so a decorator wrapped around an extractor's provider stays invisible to the trace.
• These events never reach the commit log — only recorders. They travel footprintjs's emit channel, which dispatches to recorders' onEmit and nothing else; it never writes to a stage's transaction buffer, so a resilience report can never become a CommitBundle. Inside Agent / LLMCall / Parallel that is invisible to you, because those runners attach the bridge. On a bare executor with no onEmit recorder the report is discarded outright — not stored somewhere quieter.
• A wrapper of your own that forgets to forward hooks makes everything under it go dark, silently. complete(req, hooks?) is optional in its second parameter, and TypeScript never rejects an implementation for declaring fewer parameters than its signature. So const myWrapper = (p) => ({ name: p.name, complete: (req) => p.complete(req) }) type-checks, runs, passes its tests — and swallows every report from any decorator beneath it. There is no compile error and no runtime warning to catch it. Every wrapper this library ships forwards (the three decorators plus all eight adapter wrappers), so this can only be introduced from outside. If a decorated provider reports nothing in-run, check your own wrappers first.
• Under .reliability() you will see two families interleave, and that is correct. reliability.* is the rules-based gate reporting its own dynamic decisions; error.* / fallback.* is the decorator underneath reporting its fixed-cap, exponential-backoff decisions in the same stage. Neither cross-emits the other's family — read them as two different actors, not a duplicate.

Runnable example

examples/features/35-resilience-visibility.ts walks all four cases offline — no API key, no network, hand-written providers that fail on purpose — and exits non-zero if any of them stops behaving as documented:

npm run example examples/features/35-resilience-visibility.ts

1. A dead primary, a fallback that serves
   acme-llm → backup-llm  (acme 503: gateway timeout)
   stamped at call-llm#18 in run run-1785190943438-1
   reply: Everything is up.

2. Two 503s, then it recovers
   retried  attempt 2/4 in 1ms  [http-5xx]  acme-llm 503: upstream unavailable (call 1)
   retried  attempt 3/4 in 2ms  [http-5xx]  acme-llm 503: upstream unavailable (call 2)
   recovered on attempt 3 after 6ms
   reply: Back online.

3. breaker + fallback + retry, captured by recordRun()
   fallback.triggered → error.retried → fallback.triggered → error.recovered
   reply: Served by the backup.
   16 events in the recording, all with real correlation ids
   breaker trip, visible only via the fallback's reason:
     [acme-llm] circuit breaker is OPEN — failing fast (next probe at …). Underlying error: acme 503: gateway timeout

4. Outside a run: no events at all — the hooks still fire
   breaker → open (1 consecutive failures)
   onFallback: acme 503: gateway timeout
   onRetry: attempt 2 in 1ms
   a call that passes its OWN LLMCallHooks sees: fell-back: acme-llm → backup-llm

call-llm#18 is the stage id footprintjs stamped, and run-…-1 the run — the point of emitting from inside the traversal rather than from a consumer callback, which would land the same facts under a synthetic consumer-emit#0 / consumer-scope and correlate with nothing.

Hooks for standalone use

Outside a run nothing hands the decorators an LLMCallHooks, so every report site short-circuits and no event is emitted anywhere. The optional consumer hooks are the way to observe them there — useful in a plain script or a non-agentfootprint call path. onRetry receives the upcoming attempt number and the computed backoff delay in ms:

import { withRetry, withFallback } from 'agentfootprint/resilience';

withRetry(provider, {
  maxAttempts: 5,
  onRetry: (err, attempt, delayMs) =>
    console.log(`retry ${attempt} in ${delayMs}ms: ${(err as Error).message}`),
});

withFallback(primary, fallback, {
  onFallback: (err) => console.log(`falling back: ${(err as Error).message}`),
});

The withCircuitBreaker decorator exposes an onStateChange(state, reason) hook for the same purpose. It fires wherever the breaker lives — in a run or not — which is what makes it the right seam for a cluster-wide counter; since 9.32 the same transition ALSO rides agentfootprint.error.circuit_changed inside a run, with real correlation ids. Listen to agentfootprint.cost.tick on the event dispatcher to see cost accrue across both primary and fallback providers.

A caller outside a run can also opt into the same reports the in-run call sites receive, by passing the hooks object itself. Nothing about it is private:

const ownSink: string[] = [];const hooks: LLMCallHooks = {  onResilience: (report) =>    ownSink.push(      report.kind === 'fell-back'        ? `${report.kind}: ${report.primary} → ${report.fallback}`        : report.kind,    ),};await bare.complete(oneTurn, hooks);

Unlike the in-run sink — which the library guards, so telemetry can never break an LLM call — a hook you supply is unguarded: if it throws, the throw propagates. Same contract as onRetry / onFallback / onStateChange.

Fail fast on a downed provider: withCircuitBreaker

When a vendor has a multi-minute outage, withRetry alone keeps hammering it — every request burns its full retry budget before failing over. withCircuitBreaker short-circuits that: after failureThreshold consecutive failures (default 5) the breaker OPENS and complete() throws CircuitOpenError immediately — no network round-trip — which the surrounding withFallback catches and routes elsewhere. After cooldownMs (default 30s) it half-opens and probes; halfOpenSuccessThreshold successes (default 2) re-close it.

import { withCircuitBreaker, withFallback, CircuitOpenError } from 'agentfootprint/resilience';
import { anthropic, openai } from 'agentfootprint/providers';

const provider = withFallback(
  withCircuitBreaker(anthropic({ apiKey: process.env.ANTHROPIC_API_KEY! }), {
    failureThreshold: 5,
    cooldownMs: 30_000,
    onStateChange: (state, reason) => console.warn(`anthropic breaker → ${state}: ${reason}`),
  }),
  withCircuitBreaker(openai({ apiKey: process.env.OPENAI_API_KEY! })),
);

The breaker is per-instance, not distributed — each withCircuitBreaker(...) call holds its own state in process memory. For cluster-wide coordination, layer your own Redis-backed counter on top via the onStateChange hook + shouldCount predicate.

Since 9.32 every transition is also on the typed record as agentfootprint.error.circuit_changed — { state, reason, providerName }, with the run's real runId and runtimeStageId, because a transition always happens inside a call. Before that the breaker reported nothing through the in-run channel at all, so a trip that stopped a turn was invisible on the timeline beside the tool calls it stopped.

The rules engine, same door

agentfootprint/resilience carries both halves of staying up. The four decorators above are the fixed part — retry N times, fall back, trip a breaker — and they decide the same way on every call. The reliability gate is the dynamic part: rules that read the failure and choose per call. They compose, and under .reliability() you will see both families of events interleave in one stage.

Two primitives compose ON TOP of the decorators:

• 3-tier output fallback — if both providers fail, return a canned response (or escalate).
• agent.resumeOnError(checkpoint, input) — auto-checkpoint at iteration boundaries; resume from the failure point with corrected input.

Anti-patterns

• Don't retry non-transient errors. The default policy skips AbortError + 4xx (except 429) for a reason; if you override shouldRetry, keep that distinction.
• Don't put withRetry BELOW withFallback. Wrong order: every retry on the primary delays the fallback. Right order: outer withRetry retries the WHOLE fallback chain.
• Don't compose decorators inside the provider's hot path. Build the chain ONCE at app startup; pass the composed provider into every Agent.create({ provider }).
• Don't reach for onFallback / onRetry to get telemetry into a trace. A consumer callback can only re-emit at consumer level, where the run ids are synthetic. Inside a run the events are already there.

Next steps

• examples/features/35-resilience-visibility.ts — the runnable file behind the observability section
• Error handling — typed errors + tool-error contract
• Observability — the full typed-event taxonomy these three events belong to
• Offline replay — recordRun() and rendering a saved run
• Mocks-first development — mock({ replies }) + decorators stack identically; test resilience policies offline