Executive summary
3 weeks
of prompt engineering reduced the hallucination rate less than cleaning the knowledge base did, on one project
Key conclusions
- A retrieval-augmented system can only be as accurate as the documents it retrieves. No prompt instruction can make an outdated or contradictory source say something true.
- This failure is easy to mistake for a model problem, because the output is fluent but wrong. Teams therefore focus on the model that generates the text instead of the source documents.
- High retrieval recall over a poor set of documents still produces poor answers. Remove duplicates, date every document and delete superseded material completely.
A retrieval-augmented system reads documents and answers from them. Its answers can only be as good as the documents it retrieves. No instruction in the prompt can make an outdated, duplicated or contradictory source say something true.
This failure is easy to misdiagnose because it looks like a model problem. The output is fluent but wrong, so teams focus on the model that generates the text instead of the documents that supply it.
Four data practices reduce the hallucination rate
- Remove duplicates thoroughly. When two near-identical documents differ on one figure, retrieval will return the wrong one roughly half the time.
- Date every document, and set retrieval to rank a current document above one that is relevant but out of date.
- Delete superseded material completely. Retrieval cannot treat a note saying that a document is superseded as an instruction to ignore it.
- Check retrieval quality separately from answer quality. High recall over a poor set of documents still produces poor answers.
High retrieval recall cannot make up for a weak set of documents. If the documents are low quality, accuracy falls however well the retriever performs. That makes this an integration problem first and an AI problem second.
