Memory hook: Retrieve evidence the caller may use, measure whether it is relevant, and fit it into the context deliberately.
Must remember
- Profile source formats, ownership, licences, update/delete frequency, tenant boundaries and quality before ingestion. Extract text/structure, normalise encoding, remove duplicates and keep stable document/chunk IDs with provenance. Preserve page/section references for citations.
- Chunk by semantic boundaries where practical; tune size and overlap against retrieval quality, token cost and document structure. Tables and images may need multimodal or structure-aware processing. Re-embedding requires compatible model dimensions and a planned index migration.
- Select vector storage by scale, latency, filtering, durability, tenancy, updates and operations. Approximate nearest-neighbour search trades speed against recall; metadata filters and hybrid lexical/vector search solve different retrieval failures. Reranking can improve relevance at added latency/cost.
- Enforce source/document permissions during retrieval, not by asking the model to hide forbidden information afterwards. Tenant filters must come from trusted identity context rather than an unvalidated user argument. Propagate revocations and deletion into indexes and caches.
- Separate retrieval evaluation from generation: relevant-chunk recall, ranking quality, answer relevance, groundedness and citation correctness. A plausible citation can point to a document that does not support the claim. Handle no-answer cases and conflicting/stale evidence explicitly.
- Context budgets include instructions, conversation, retrieved chunks, tool results and output allowance. Summarise or select context with traceable rules; avoid truncating crucial constraints. Prompt templates, model/embedding versions and retrieval settings belong in release metadata.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| Correct document exists but ranks too low | Inspect embedding/query representation, filters, hybrid search and reranking. |
| Only one tenant should see a document | Enforce trusted tenant/ACL filters before model context construction. |
| Index must change embedding dimensions | Build and evaluate a compatible new index, then migrate traffic. |
Traps
- Vector similarity is not factual entailment.
- Document deletion without index/cache invalidation can preserve access.
- A citation needs support validation, not just a valid URL.
Active recall
1. Why keep stable chunk IDs?
To update/delete content deterministically, trace provenance and compare retrieval results.
2. What is the trade-off of more overlap?
Potentially better boundary coverage but more storage, tokens, duplicate context and cost.
3. Where should tenant access be enforced?
In trusted application/retrieval controls before evidence reaches the model.
4. Which failure suggests generation rather than retrieval?
The correct evidence is present but the answer misstates or ignores it.
5. How should unsupported questions be handled?
A defined abstention/escalation path, rather than fabricating an answer.