A model can retrieve an excellent explanation from a knowledge base and still give the wrong answer about a fact that changed five minutes ago.
Retrieval is useful because it gives a model relevant material at the moment of a question. A policy, a manual, or a catalogue of approved guidance can be indexed, searched, and supplied as context. This is an effective shape when the answer lives in reference material that remains materially true between index refreshes.
The same mechanism is often stretched into a job it cannot own: answering questions about a current order, an available stock level, an open ticket, or an account change. Those are not simply documents that happen to be retrieved. They are live values, owned by a transactional system that can change independently of an index. Treating them as text snapshots creates a subtle failure: the model receives real-looking material, produces a fluent answer, and has no reliable way to know whether that material is still current.
The boundary is architectural, not cosmetic. A smaller chunk, a newer embedding model, or a more frequent refresh does not turn an index into the authority for a value it does not own. The first decision is to identify the kind of information a question needs. The second is to route that question to the mechanism whose relationship with that information is appropriate.
Policies, manuals, FAQs, and reference material can be retrieved as relevant context.
Current values change outside the index and remain owned by a transactional system.
Retrieve stable knowledge; call the system of record for current facts.
Route each request deliberately, then assess each architecture against its own evidence.
Start by naming the kind of information
A knowledge base is a collection of reference material. It may contain product documentation, internal procedures, policy language, or instructions that give a reader durable context. When someone asks what a rule says or how a process works, retrieval can locate the relevant passages and give the model grounded material from which to respond.
Live state has a different property: its current value is maintained by a system outside the index. An order may move, stock may change, a ticket may be assigned, or a customer record may be updated after an earlier snapshot was written. The fact is not merely stored somewhere; it is actively owned and changed by a system of record. Its authority comes from that system's current value, not from the semantic resemblance of an older description.
That distinction is easy to blur because both can arrive at a model as text. A retrieved passage may state that an item was dispatched; a service can return the current position of the same item. The surface form is similar, but the operating meaning is not. One is evidence about a durable reference. The other is a request for a present value. Architecture should follow that difference before a prompt is written or an index is tuned.
See the category error before it looks like a model error
Consider a hypothetical customer-support assistant. It retrieves a help article explaining returns and two historical notes about a customer's shipment. One note says the order was dispatched; another says it was being prepared. Both were accurate when they entered the knowledge corpus. Since then, the package was redirected and is awaiting a new dispatch. That current change exists in the order service, not in either retrieved snapshot.
The model receives the passages that are most similar to the question and produces a coherent update. Nothing necessarily crashes. Latency can look normal. The retrieved text can be genuine. Yet the response is wrong because the index contains competing historical descriptions rather than the current value. The failure may remain invisible until the customer notices that the answer conflicts with reality.
This is why similarity is not truth. A strong match tells the system that a passage resembles the query. It does not establish that the passage is the latest authoritative state. Increasing similarity thresholds cannot answer a question the underlying source cannot resolve. Nor does a shorter refresh interval remove the category error: between any two refreshes, live state can still change.
Route each question to the owner of the answer
The repair is not a better retrieval setting. It is a tool call to the system that owns the live value. In the hypothetical support flow, a documentation question can go to retrieval; a request for the current order position goes to the order-status service. The model can use the returned value to explain it in plain language, but it should not substitute a retrieved historical statement for the current fact.
That routing can sit within a larger workflow. A request may first be classified by what the person is asking for, then sent to a knowledge layer for a policy explanation, a transactional API for current state, or a human review path for an action with significant consequences. Combining architectures can be appropriate when the parts of the problem fail in genuinely different ways. It is not a reason to add mechanisms before the problem has been named.
| Question needs | Appropriate mechanism | What the response can rely on |
|---|---|---|
| Stable policy, manual, or FAQ | Retrieve relevant passages from the knowledge corpus. | Reference material that remains useful between index refreshes. |
| Current status, balance, availability, or queue | Call the business system that owns the live value. | The present value returned by the system of record. |
| High-consequence change or exception | Route to the designated human approval path. | The current state, the proposed action, and the accountable decision. |
Each route should be evaluated according to the work it owns. A healthy top-level experience can hide a failure in one component: retrieval may answer policy questions well while a live-state route returns incomplete values, or the reverse. Separate evidence for separate mechanisms prevents a combined system from looking successful merely because one path happens to be strong.
Test the boundary, not only the response
Reference architectures are useful because they show recurring shapes and their known failure modes. They are not blueprints to copy without judgment. A durable design adapts the appropriate shape to the workload, then makes the boundary between shapes inspectable. In this case, an operator should be able to see whether a request was treated as stable knowledge or live state, which mechanism served it, and where the returned information came from.
Begin with the questions most likely to expose the distinction. Ask the knowledge path about a policy that is present in the corpus. Ask the live path about a value that changes after an earlier snapshot exists. Check that the first reaches relevant reference material and that the second reaches the system of record. The goal is not to make one general demonstration look persuasive. It is to verify that each mechanism is doing the job it was selected to do.
If a request combines both types of information, keep their roles explicit. A support response may retrieve the relevant returns policy and call the order service for the current shipment state. The reader then receives both a grounded explanation and a current fact, without asking either data source to impersonate the other. This is simpler to reason about than a single corpus expected to answer every question.
An architecture test for every question
Before sending a new question through retrieval, ask:
- Is the answer a stable piece of reference knowledge, or a value that changes in a live business system?
- Which system owns the current fact, and can the workflow call it directly?
- Would an older but semantically relevant snapshot be unsafe to present as the answer?
- If the request needs both policy and current state, which route supplies each part?
- What evidence will show that each route worked for the question type it owns?
- Which actions or exceptions need a human approval path rather than an automated response?
A knowledge base is valuable precisely because it does not have to be a live system of record. Let it hold the stable material it is designed to make available. Then call the system that owns current facts when the question is about now. That boundary prevents a polished answer from becoming a confident account of the past.
Reliable AI starts by asking which system has the right to answer the question.
Back to all field notes