Names and numbers from evidence
Every number, identifier and name in the final answer must appear in a tool result the run really read. If one does not, the model typed it rather than read it — a deterministic check, no second model.
A storage engineer asks which array port is affected. The agent answers with a port row: alias
SHPMAXDLVAP001-FA0, FCID0xef0101. Both look exactly like the real thing. Neither appears in any tool result from that turn — the model typed them. That is a real answer from a real production run, and nothing in the conversation could have caught it. The framework watched every tool result land, so it can.
.namesAndNumbersFromEvidence() requires every number, identifier and name
in the final answer to appear in a tool result the run actually read.
const agent = Agent.create({ provider, model })
.tool(showInterfaceStatus)
.tool(showFlogi)
.namesAndNumbersFromEvidence({ posture: 'guard' })
.build();
const answer = await agent.run({ message: 'which array port is affected?' });What it is — and what it provably is not
It is a fabrication detector, not a correctness judge.
It catches invented values. It cannot catch a false claim assembled from
real values: "fc1/3 is healthy" when the data says the port is down uses
entirely grounded tokens — fc1/3 is in the evidence, "healthy" is a word — and
the check passes it without a murmur. So does "the outage started at 08:15"
when 08:15 is a timestamp belonging to a different port. If you read this as a
hallucination check you will trust it for the one thing it cannot do.
It is also conservative in the other direction. Small numbers, all-letters names
and quantities with units (32G, 47 flaps, 892 CRC errors) are not
examined, because a false accusation costs a real turn — and on a weak model
a false accusation triggers exactly the retry loop this library exists to
remove. A missed fabrication is a miss; a refused good answer is damage.
Why it is deterministic
No second model, no embedding, no LLM judge. The check is set membership over normalized tokens: it costs microseconds and answers identically on every run.
That constraint is the point. This library's thesis is that structure lets a smaller model perform like a bigger one — so a guard that needed a bigger model to police the small one would invert the whole value proposition, and would fail precisely where the small model is deployed: offline, cheap, fast.
The three postures
Same three words as .skillGraph({ strictness })
and a separate option, because routing authority and evidence discipline are
different decisions.
posture | What happens |
|---|---|
'assist' (default) | Record and flag. The answer goes out unchanged — you learn how often it happens before you act on it. |
'guard' | The unsupported values are named back to the model, which gets one more ordinary turn. Survivors ship flagged. Recommended for weaker models. |
'rails' | The same one revision, then run() raises UnsupportedValuesError rather than return an answer that still carries them. |
guard is a branch of the ReAct loop, not a special mode: the correction is one
more ordinary turn, with its own iteration_start / llm_start bracket and its
own cost.tick — and the tools are still on the wire, so the model can go and
fetch the value it guessed at. One revision per turn, latched: a model that
cannot ground a value on its second try will not on its fifth.
// guard, in the run's own record:
// route_decided { chosen: 'evidence-recheck' }
// evidence_checked { action: 'revision-asked', unsupported: [ … ] }
// route_decided { chosen: 'final' }
// evidence_checked { action: 'grounded', afterRevision: true } ← it workedWhat counts as evidence
- Tool results — the
role: 'tool'turns of the conversation. That is the whole corpus. Results that are JSON are walked structurally (every key, every leaf) rather than searched as text, so a value that merely appears inside an unrelated field does not read as grounded. - Tool call arguments do not count. The model typed those; grounding a value because the model passed it to a tool would let any invention launder itself through one failed lookup.
- The model's own earlier answers do not count, for the same reason.
Values the user supplied are exempt without being declared: this turn's message, the conversation's user and system turns, and the composed system prompt (base prompt, skill bodies, retrieved passages). The user gave them, so they were not invented.
Spelling differences are normalised on both sides, which is where a naive
implementation produces its worst false positives: 41,200 in prose matches the
JSON number 41200, 2048.0 matches 2048, and 0xef0101 matches ef0101.
Teaching it your identifiers
The default extractor guesses from digits and punctuation. It cannot know that
ORD-4471 is an order number or that a WWN is a WWN. Declare the shapes and
they are checked by name — shapes composes with the defaults rather than
replacing them, and exempt removes what you know is safe.
.namesAndNumbersFromEvidence({
posture: 'guard',
shapes: [
{ name: 'wwn', match: /(?:[0-9a-f]{2}:){7}[0-9a-f]{2}/ },
{ name: 'order', match: /ord-\d{4,}/ },
],
exempt: ['v9.35.0', /^build-\d+$/],
minDigits: 4, // when a BARE number stops being prose. Default 4.
})A NamesAndNumbersOptions bag: posture (EvidencePosture), shapes
(EvidenceShape[] — each a name plus a match pattern, anchored to a whole
token), exempt, and minDigits.
Reading the record
Every judgement fires agentfootprint.agent.evidence_checked — whatever the
posture and whatever the outcome, so a debugger can show the answer, the values
and whether the revision fixed them.
agent.on('agentfootprint.agent.evidence_checked', (e) => {
// action: 'grounded' | 'revision-asked' | 'flagged' | 'refused'
log.info({ action: e.payload.action, values: e.payload.unsupported });
});After the run, agent.unsupportedValues() returns the terminal verdict — the
flagged UnsupportedValue[], whether a revision was spent (revised), and
whether the answer was withheld (refused) — or undefined when every value
was grounded.
Under 'rails' the run raises UnsupportedValuesError (an
UnsupportedValuesContext: the values, the candidate count, revised). The
error names the values and says what would satisfy the check; the refused
answer is not carried on it and stays in the commit log under whatever
redaction the run configured.
try {
await agent.run({ message: 'which array port is affected?' });
} catch (e) {
if (e instanceof UnsupportedValuesError) {
log.warn({ invented: e.values.map((v) => v.value) });
} else throw e;
}The correction the loop sends is an authored frame followed by the quoted
values; EVIDENCE_CHECK_FRAME_PREFIX is exported so a transcript reader (or a
test) can recognise it without matching on prose.
Composition
- With
.outputSchema(): the schema is judged first — an answer with the wrong shape is about to be replaced wholesale, so grounding it would pay for the same turn twice. The evidence gate judges what the schema let stand, and an answer that exhausted its schema retries is not judged at all. - With
.reliability(): no collision. Reliability governs what one call does before a response is committed; this governs an answer that was committed. - Unused, it costs nothing: an agent without the option mounts no branch, writes no state, and emits no event.
When to reach for it
Reach for 'assist' on any agent whose answers carry identifiers a human will
act on — port names, order numbers, account ids, serials — and read the events
for a week. Move to 'guard' when the numbers say you should, and especially
when the model is a small one: naming the values back is the cheapest structural
help a weak model gets. Reach for 'rails' only where an answer carrying an
invented identifier is worse than no answer at all.
