Observability

Logs that say "agent ran" are useless. Logs that say "agent called redact_pii({text: '...'}) and then injected the result into the messages slot before iteration 3, costing $0.0042" let you debug a hallucination at 2 AM. The difference is whether observability is structured + typed at the source or string-formatted at the sink.

Three workflows from one trace

agentfootprint's observability isn't bolted on. The framework owns the loop, so every decision and execution is recorded during DFS traversal — not collected after the fact via spans. That single artifact serves three workflows:

And a fourth, novel: the agent can read its own trace. Six months after the agent rejected loan #42, "why did you reject it?" answers from the recorded evidence (creditScore=580, threshold=600), not a rerun. Causal memory turns the trace into the agent's working memory.

The model

Every runner (Agent, LLMCall, Sequence, etc.) exposes the same observability surface, in three layers:

There are 92 typed events across 21 domains: agent (incl. output_schema_validation_failed for Instructor-style retry and output_schema_retry for the loop-level re-ask), composition, context, stream, tools, skill, memory, cost, permission (incl. halt for sequence governance), credential (declare-and-push resolution), artifacts (the claim-check lifecycle: minted / resolved / expired / refused / presented), eval, embedding, pause, checkin (evidence-carrying human consent), middleware (a governance chain answered), error, reliability, fallback, risk. Every event has a typed payload — agentfootprint.stream.llm_start, agentfootprint.stream.tool_start, agentfootprint.cost.tick, agentfootprint.composition.iteration_start, etc. The framework owns the loop and emits during DFS traversal — no instrumentation, no spans-after-the-fact.

Tier-3 — .enable.liveStatus() and .enable.observability()

The 80% case: a live status line ("what is the agent doing right now?") and a domain-filtered log firehose. Both are one-liners over a strategy — pick a built-in (chatBubbleLiveStatus, consoleObservability from agentfootprint/observe) or a vendor one.

// Live status line — user-facing "what's the agent doing right now".// The chat-bubble strategy maps the thinking-state machine to one line.const stopThinking = agent.enable.liveStatus({  strategy: chatBubbleLiveStatus({ onLine: (line) => console.log(`  ⎈ ${line}`) }),});

.enable.liveStatus({ strategy }) with chatBubbleLiveStatus({ onLine }) gives you a Claude-Code-style live status line — one string per agent state transition. Use for terminal UIs, CLI progress, or chat-UI typing indicators.

// Firehose observability via the console strategy. Swap in any vendor// strategy (Datadog, OTel, AgentCore, CloudWatch) — same call site.const stopLogging = agent.enable.observability({  strategy: consoleObservability({ logger: { log: (...args) => console.log('  [log]', ...args) } }),});

.enable.observability({ strategy }) with consoleObservability() gives you structured logs of every typed event. Swap the strategy for pino, winston, or a vendor backend — same call site.

Both return a stop() function — call it on shutdown to detach the listener cleanly.

Migration (4.0.0): the old flat .enable.thinking() / .enable.logging() one-liners were removed in favor of these uniform strategy enablers. Replace enable.thinking({ onStatus }) with enable.liveStatus({ strategy: chatBubbleLiveStatus({ onLine: onStatus }) }), and enable.logging() with enable.observability({ strategy: consoleObservability() }).

Tier-3 — .enable.observability() — grouped vendor strategies

For shipping events to a vendor backend (CloudWatch, X-Ray, OTel, AgentCore), .enable.observability(...) accepts a typed strategy + optional detach driver. The strategy is the WHERE; the detach is the HOW (sync inline vs fire-and-forget):

import { agentcoreObservability } from 'agentfootprint/observe';
import { microtaskBatchDriver } from 'footprintjs/detach';

const telemetry = agent.enable.observability({
  strategy: agentcoreObservability({
    region: 'us-east-1',
    logGroupName: '/agentfootprint/my-agent',
  }),
  // Recommended for production — slow exporters never block the agent loop.
  // The strategy's `exportEvent(event)` runs on the driver's schedule
  // instead of inline on the dispatcher's hot path.
  detach: { driver: microtaskBatchDriver, mode: 'forget' },
});

What comes back — ObservabilityHandle

telemetry is an ObservabilityHandle (StrategyHandle is the same type under its group-neutral name — enable.cost() and enable.liveStatus() return it too). It is still the unsubscribe function it has always been, so every call site written before 8.12.0 keeps working:

Where your runtime has Symbol.asyncDispose, the handle is an AsyncDisposable, so await using telemetry = agent.enable.observability({ strategy }) flushes and releases at the end of the scope.

agent.run() never flushes — telemetry must not become a term in run latency. For scripts and short-lived processes, opt into a per-run drain with flushOn: 'run-end', which FIRES a flush when a run ends without gating run() on it.

Detach modes

Drivers

Pick by environment — see footprintjs/detach for the full list. All exported from 'footprintjs/detach':

microtaskBatchDriver, setImmediateDriver, and setTimeoutDriver ship as ready-to-use singletons. createSendBeaconDriver and createWorkerThreadDriver are factories (no default singleton) because they need required options — call the factory and pass the result as driver.

Graceful shutdown

Since 8.12.0 the handle enable.observability() returns drains itself — and it is the only thing that can, because a detached event is sitting on the driver, not in your strategy's buffer:

const telemetry = agent.enable.observability({
  strategy,
  detach: { driver: microtaskBatchDriver, mode: 'forget' },
});

await telemetry.flush();   // driver queue first, then the strategy's buffer
await agent.shutdown();    // …or drain and release everything in one call

handle.flush() waits for the events THIS subscription scheduled. flushAllDetached() (from 'footprintjs/detach') remains the process-wide hammer for detached work agentfootprint did not schedule.

Vendor adapters — agentfootprint/observe

Vendor strategies ship under one grouped subpath (parallel-providers pattern, mirrors llm-providers / tool-providers / memory-providers):

All peer deps are declared optional — consumers who never call a particular factory don't need to install its SDK. Each adapter lazy-imports its SDK at first use.

enable.cost(...) accepts the same detach option for the same reasons. enable.liveStatus and enable.flowchart deliberately stay sync — UI render must feel responsive.

Cost tracking

Pass a pricingTable to the Agent constructor and the framework emits agentfootprint.cost.tick after every LLM call with per-call and cumulative USD. Add a costBudget and you also get a one-shot cost.limit_hit the first time cumulative crosses the budget.

A bare number warns: the event fires with action: 'warn' and the run carries on, and you decide what to do (cancel via signal, escalate, log). Since 8.14.0 you can also ask it to stop:

Agent.create({ provider, model, pricingTable, costBudget: 0.50 })                      // warns
Agent.create({ provider, model, pricingTable,
               costBudget: { usd: 0.50, onExceed: 'halt' } })                          // stops

'halt' ends the loop at the next iteration boundary — the same boundary maxIterations uses. Never mid-call: the call that crossed the budget completes, is billed and is recorded; halting only decides there will not be another one. The event then reports action: 'abort', and agent.stoppedEarly() says what happened:

const answer = await agent.run({ message });
const cut = agent.stoppedEarly();
if (cut) console.log(`stopped at iteration ${cut.iteration}: ${cut.reason}`);

// The same budget, made a STOP. Halting ends the loop at the next iteration// boundary — the same boundary maxIterations uses — so the call that crossed// the budget still completes, is still billed and is still recorded. What// halting decides is that there will not be another one.const halting = Agent.create({  provider: provider ?? exampleProvider('feature'),  model: 'demo-sonnet',  pricingTable: pricing,  costBudget: { usd: 0.00001, onExceed: 'halt' }, // crossed on the FIRST call})  .system('')  .tool({    schema: { name: 'noop', description: '', inputSchema: { type: 'object' } },    execute: () => 'ok',  })  .build();const halted = await halting.run({ message: 'do the thing' });const cut = halting.stoppedEarly();if (cut) {  // `AgentOutput` is a bare string, so the reason cannot ride the answer. It  // rides committed state instead, where it is provable after the run rather  // than only observable by whoever happened to be subscribed.  console.log(    `\nhalted at iteration ${cut.iteration} (${cut.reason}); ` +      `${cut.pendingToolCalls} tool call(s) never ran; ` +      `answer was ${cut.answerWasEmpty ? 'EMPTY' : 'partial but real'}`,  );} else {  console.log('\nthis run finished before the budget was crossed:', halted);}

stoppedEarly() also covers maxIterations. It is set only when a limit cut the turn short — the model was still asking for tools and the run refused to run them — so a turn that used its whole budget and then genuinely finished reports nothing. When the answer that comes back is the empty string the library warns once on the console too, because an empty string reaching a user is indistinguishable from a bug.

// 'feature' kind drives the smart tool-call flow. Cost ticks fire// automatically off the per-iteration usage MockProvider estimates// (chars/4) — sufficient to demo the budget crossing.const agent = Agent.create({  provider: provider ?? exampleProvider('feature'),  model: 'demo-sonnet',  pricingTable: pricing,  costBudget: 0.0001, // trip the warning})  .system('')  .tool({    schema: { name: 'noop', description: '', inputSchema: { type: 'object' } },    execute: () => 'ok',  })  .build();agent.on('agentfootprint.cost.tick', (e) => {  const p = e.payload;  console.log(    `[tick] +$${p.estimatedUsd.toFixed(6)} — cumulative $${p.cumulative.estimatedUsd.toFixed(6)}`,  );});

Event taxonomy

agentfootprint.<domain>.<event>

Every event is exhaustively typed via AgentfootprintEvent (a discriminated union); your IDE autocompletes the payload shape based on the event name. No string-parsing logs. No "I think this means the tool was called."

Custom aggregations

For aggregations the built-ins don't cover, subscribe to the typed events directly. runner.on(type, fn) returns an unsubscribe — the payload is fully typed by the event name:

let injections = 0;
let totalOutputTokens = 0;

const stopA = runner.on('agentfootprint.context.injected', (e) => {
  // e.payload — { slot, source, contentSummary, asRole?, ... }
  // `asRole` is a `ContextRole` — 'system' | 'user' | 'assistant' | 'tool',
  // the role the piece carries in the conversation. Present on messages-slot
  // records (which report the conversation itself), absent on system-prompt ones.
  injections++;
});

const stopB = runner.on('agentfootprint.agent.turn_end', (e) => {
  // e.payload — { turnIndex, finalContent, iterationCount, totalInputTokens, totalOutputTokens, durationMs }
  totalOutputTokens += e.payload.totalOutputTokens;
});

// Or subscribe to a whole domain with a wildcard:
const stopC = runner.on('agentfootprint.context.*', (e) => { /* every context.* event */ });

// On shutdown:
stopA(); stopB(); stopC();

On context.injected, asRole is a ContextRole — 'system' | 'user' | 'assistant' | 'tool', the role the piece carries in the conversation. It is present on messages-slot records, which report the conversation itself, and absent on system-prompt records, which have no message role to carry. The same union types LLMMessage.role, so a consumer grouping injections by role and a consumer reading the request are speaking about the same four values.

runner.attach(recorder) is the lower-level path: it accepts a footprintjs CombinedRecorder that observes the raw footprintjs stream (onWrite, onSubflowEntry, onDecision, …) — not the grouped agentfootprint.* events. For agentfootprint's typed event taxonomy, prefer runner.on(...). A CombinedRecorder discovers its handler methods via runtime method-shape detection — implement only the hooks you care about; ignore the rest.

OpenTelemetry

otelObservability ships in agentfootprint/observe. It maps agentfootprint events onto OTel spans following the GenAI semantic conventions (gen_ai.* attributes) via your own @opentelemetry/api tracer. Pair with @opentelemetry/sdk-node and your favorite exporter (Jaeger, Tempo, Datadog OTel collector, etc.):

import { otelObservability } from 'agentfootprint/observe';
import { microtaskBatchDriver } from 'footprintjs/detach';
import { trace } from '@opentelemetry/api';

const otel = otelObservability({
  serviceName: 'my-agent',
  tracer: trace.getTracer('my-agent', '1.0.0'),
  // genAiSpanNames: true,  // opt-in spec span names ('chat gpt-4', 'execute_tool search', …)
});
const stop = agent.enable.observability({
  strategy: otel,
  detach: { driver: microtaskBatchDriver, mode: 'forget' },
});

Span tree — one trace per turn: invoke_agent root (agent name, turn-total gen_ai.usage.* tokens) → iteration:N → chat spans per LLM call (gen_ai.provider.name, gen_ai.request.model, usage incl. cache tokens, gen_ai.response.finish_reasons) and execute_tool spans per tool call (gen_ai.tool.name, gen_ai.tool.call.id, parallel-safe correlation).

Explainability span events (on by default; explainability: false to opt out) carry the decision trail a compliance reviewer needs: route decisions, skill-routing provenance (decision path + unlocked tools), tool-arg validation rejections, permission checks/halts, credential lifecycle. PII discipline mirrors the validation contract — tool args appear as key names, results as a type, prompts and LLM content never.

For operator-level decide()/select() evidence (creditScore gt 700 → 750 (true)), attach the strategy's FlowRecorder bridge too:

const agent = Agent.create({ provider, model })
  .watch(otel.decisionEvidenceRecorder()) // decide()/select() evidence → span events
  .build();

Runnable demo: examples/features/18-otel-genai.ts.

End-to-end compliance lighthouse — spans + tamper-evident audit export + causal memory feeding an OFFLINE auditor that explains a loan decline weeks later from persisted JSON alone: examples/features/20-regulated-decisioning.ts.

Anti-patterns

• ❌ Don't log every event — dozens of typed events × N iterations × M agents = your log infra hates you. Filter to the domains you actually need.
• ❌ Don't subscribe in a hot loop — runner.on(...) returns an unsubscribe; call it on shutdown. Forgotten subscriptions leak listeners across runs.
• ❌ Don't use getNarrative() as a log sink — it's structured trace data, not log lines. Pass the entries to a real log/trace backend if you need to ship them.

Next steps

• Localize a context bug (Beta) — when the trace shows what happened but not which input caused it
• Causal memory deep dive — the trace as the agent's working memory
• Reliability gate — agentfootprint.reliability.fail_fast event semantics
• Streaming guide — token-by-token rendering via provider.stream()

Embedding cost — embeddingRecorder

embeddingRecorder (with EmbeddingRecorderOptions) is the always-on bridge that forwards agentfootprint.embedding.generated to the typed stream. It is attached on every run, so you only construct one yourself when wiring a recorder outside Agent.

The event separates the two halves of embedding cost through inputKind: 'document' is index time (once, scaling with corpus size) and 'query' is retrieval time (per turn, scaling with traffic).

agent.on('agentfootprint.embedding.generated', (e) => {
  console.log(e.payload.inputKind, e.payload.count, e.payload.durationMs);
});