Indexing a corpus

You have a folder of documents and you want an agent to answer from it. Before 8.10.0 the library gave you the second half of that — defineRAG retrieves, sqliteVectorStore keeps — and left you to write the loaders, the splitters and the re-index logic yourself. This is the first half.

The three steps, and the one call over them

import { indexFolder } from 'agentfootprint/rag';
import { sqliteVectorStore } from 'agentfootprint/memory';
import { staticEmbedder } from 'agentfootprint/providers';

const report = await indexFolder('./docs', {
  to: sqliteVectorStore({ file: './corpus.db' }),
  embedder: staticEmbedder(),
});
// { discovered: 3, loaded: 3, chunks: 14, embedded: 14, skipped: 0, removed: 0, … }

indexFolder is sugar. The pieces are public and compose in the obvious order:

import { loadDocuments, splitDocuments, byHeading, indexCorpus } from 'agentfootprint/rag';

const { documents, failed } = await loadDocuments({ dir: './docs' });
const chunks = splitDocuments(documents, byHeading());
// …or hand the whole thing to the chart, which does both and records what it did:
const report = await indexCorpus({ source: { dir: './docs' }, store, embedder });

Where the index lands

store is any vector-ranking MemoryStore, and the indexers are identical across all of them — swapping one for another changes durability and operations, not a line of the call:

import { sqliteVectorStore, pgVectorStore, s3VectorsStore } from 'agentfootprint/memory';

sqliteVectorStore({ file: './corpus.db' });                          // one machine, one file
pgVectorStore({ client: pool });                                     // beside your own data
s3VectorsStore({ bucket: 'my-corpus', index: 'docs' });              // nothing to run

The last one is the interesting change of shape: an index in S3 Vectors can be added to from a cron job at 14:00, and the agent sees it on the next turn. An index in a corpus bundle changes when you redeploy.

A store that declares it cannot serve vectors back is refused before anything is embedded, and therefore before anything is billed — including a store that ranks TEXT on its own server (ranksBy: 'server-text'), whose own console does its ingestion. See ranking mode.

Loaders

loadDocuments routes by extension. A loader you pass is consulted before the built-ins, so overriding one is putting yours in front rather than editing a registry:

await loadDocuments({ dir: './docs' }, { loaders: [myConfluenceLoader()] });

Markdown keeps its markup. Stripping # would throw away the one splitting signal that is not a heuristic — the author already told us where the sections are — and rewriting text would break the offsets every citation depends on.

HTML is a tag stripper, not a parser. Good enough for documentation pages and exported articles; not good enough for a single-page app, where you will get navigation labels. Run a real extractor and feed the result in through { text, uri }.

PDF needs one optional peer. unpdf was picked by measurement — 2.5 MB and zero transitive dependencies, against 86 MB and a native binary for the nearest alternative. It returns text per page, which is why a PDF citation can name a page you can turn to. Missing, it refuses with an install line rather than skipping the file.

Splitters

Defaults are 1000 characters with 150 of overlap, and they are chosen by constraint rather than taste. localEmbedder's default model silently truncates at 512 wordpiece tokens — measured directly: at 508 tokens an appended tail still moves the vector, at 596 it does not, and nothing says so. 1,000 characters is about 250 tokens, comfortably inside that cliff and inside the length the model was trained at.

The floor — minChars (8.20.0)

Short chunks retrieve too well, not too badly. Similarity is a density measure: a heading plus one preamble sentence concentrates its topic's vocabulary with none of its substance. Measured in a production corpus, a 180-character heading-and-preamble chunk outranked the 1,032-character body of its own section — and the model, handed a passage that promises findings and contains none, fabricated a plausible file path to fill the gap.

So byHeading and byParagraph enforce a floor. A section (or packed paragraph chunk) whose own text is under minChars — default min(250, maxChars / 4) — merges forward into the next chunk under its own heading: the preamble sentence survives, leading the chunk it introduces, and nothing is ever dropped. The last chunk has no next, so a trailing short merges backward. Two rules are unconditional, even at minChars: 0: a heading with no body is never emitted alone (a coordinate, not a passage), and a document that is nothing but headings yields no chunks at all. fixedWithOverlap is exempt by design — its chunks are uniformly sized by request — and wholeDocument is one chunk per document by definition.

Re-indexing an existing corpus after upgrading will produce different chunks. That is the fix, not a regression: the old default could ship the exact chunk shape that drove a fabricated citation.

The ceiling — maxChunkChars and maxInputChars (9.1.0)

Two numbers bound one chunk, and they are set in different places. maxChars on the splitter decides how big a chunk is cut. maxChunkChars on indexCorpus / indexFolder / indexDocuments decides how much of it the embedder actually reads. At the shipped defaults they compose safely: 1,000 characters cut, at least 2,000 read.

The trap opens when you raise the splitter's ceiling. byHeading({ maxChars: 2500 }) against a 2,000-character reading ceiling produces chunks that are stored whole as the passage and indexed by their opening. Retrieval then cannot find wording that is plainly visible in the <source> block the model is shown, and nothing throws. Measured in a production corpus: 6 of 26 chunks, against an embedder that would have read every one of them in full.

So the reading ceiling comes from the embedder, which is the only object that knows it:

An explicit maxChunkChars on the call always wins over the embedder's number: dense text (code, tables, CJK) tokenises tighter than 4 characters a token, and you are allowed to know that about your own corpus.

Chunks past whichever ceiling is in effect are recorded, and — since 9.1.0 — said out loud once per run, because a number nobody reads is not visibility:

report.truncated; // [{ id: 'handbook.pdf#3', chars: 4200 }]
report.truncatedCount; // 1  ← the number you can assert on

indexCorpus: 6 of 26 chunk(s) are longer than the 2000-character input ceiling in effect
  ('local:Xenova/all-MiniLM-L6-v2:q8's declared maxInputChars), so they were embedded CLIPPED.
  The vector represents only the opening of each one, while the FULL text is stored and served
  as the passage — so retrieval cannot find wording that is plainly visible in the block the
  model is shown. …
  Fix:  re-split smaller (lower the splitter's maxChars), or raise maxChunkChars if … really
  reads more than 2000 characters.

Writing your own Embedder? Declare maxInputChars on it. It is optional — absent, an indexer behaves exactly as it did before the field existed — but the alternative is that your ceiling is discovered by someone reading a retrieval result that is missing a paragraph they can see.

The invariant every splitter holds

doc.text.slice(chunk.charStart, chunk.charEnd) === chunk.text

splitDocuments checks it rather than trusting it, and refuses a splitter that breaks it. A chunk that cannot be located in its own document produces citations that point at the wrong words — and a citation nobody can check is worse than no citation, because it looks checked.

indexCorpus — the chart

Indexing is a footprintjs chart, not a loop:

discover → load → split → plan (DECIDER)
         → take-window → embed (FAN-OUT + retry) → tally → more? ⟲
         → remove → report

The commit log IS the indexing report. A for loop answers no questions after it finishes; the chart answers "why is this chunk here, why was that one skipped, what did this run cost, which document went missing" months later, from its own log, without the caller having saved anything.

• plan is a real decider because "skip this chunk" is a decision with evidence: same content hash and same embedder fingerprint. Its branches — full-index, incremental, nothing-to-do — are named in the trace.
• embed fans out one branch per batch with a declarative retry, so a rate-limited attempt appears in the record instead of vanishing inside a hand-rolled loop. It runs a window at a time and loops: the fan-out ceiling truncates rather than queues, so a single fan-out over 200 batches with a ceiling of 4 would embed 4 and report success.
• embedded is counted from what the batches actually wrote, never from what the plan intended.

A failing batch fails the run. A half-indexed corpus keeps answering and quietly cannot see what did not land, which reads as "the model does not know that" rather than as a failure.

Incremental re-index

Run it again and nothing happens, which is the point:

run 1  (first index)   discovered 3 · loaded 3 · chunks 8 · embedded 8 · skipped 0 · removed 0
run 2  (no changes)    discovered 3 · loaded 3 · chunks 8 · embedded 0 · skipped 8 · removed 0
run 3  (edit + delete) discovered 2 · loaded 2 · chunks 5 · embedded 1 · skipped 4 · removed 3

A chunk is reused when its content hash and the embedder fingerprint both match. Change the text and only that chunk is re-embedded; change the embedder and everything is, because the two vector spaces are not comparable. Delete a document and its chunks are removed — an index that still answers from a file you deleted is worse than one that cannot answer.

An empty walk never prunes. A typo in a path must not delete a corpus.

From the command line

npx agentfootprint-index ./docs --to ./corpus.db
npx agentfootprint-index ./docs --to ./corpus.db --embedder local --split paragraph --chars 800
npx agentfootprint-index ./docs --to ./corpus.db --dry-run --json

--embedder is static (default, no key, no network), local, openai, or mock. --dry-run reports against a throwaway in-memory index without touching the file.

The worked example

const index = () => indexFolder(docs, { to: store, embedder, embedderId: embedder.id });// RUN 1 — everything is new.const first = await index();lines.push(summarize('run 1  (first index)   ', first));// RUN 2 — nothing changed. Nothing is embedded again.const second = await index();lines.push(summarize('run 2  (no changes)    ', second));// RUN 3 — edit one document, delete another.const policy = join(docs, 'refund-policy.md');writeFileSync(  policy,  readFileSync(policy, 'utf8').replace('within 3 business days', 'within 5 business days'),);unlinkSync(join(docs, 'pricing.md'));const third = await index();lines.push(summarize('run 3  (edit + delete) ', third));

Three real documents ship with it — two Markdown and a two-page PDF — and the agent's answer comes back with the passage, the document, the page and the score:

why this passage
  ✓ refund-policy.md#1           0.87  refund-policy.md, Refund timing
  ✓ refund-policy.md#3           0.86  refund-policy.md, Partial refunds
  ✗ security-overview.pdf#0      0.82  security-overview.pdf, p1  over-max-entries

Reference

Everything on agentfootprint/rag.

Pipeline

Value objects

Ports and adapters

Deliberately not here

• Re-ranking and MMR are retrieval-side. They belong behind RetrievalStrategy on agentfootprint/memory — a re-ranker is a different rule for choosing among candidates, not a different way of building an index.
• File watching. indexCorpus is an explicit call: run it at boot, from a cron, or from the CLI. Magic that re-indexes behind you is magic that re-bills you.

Next steps

• RAG — defineRAG, and reading back why a passage was retrieved
• Memory store adapters — sqliteVectorStore, pgVectorStore, s3VectorsStore, and what each store's search() ranks