Good RAG Answers Questions. Great RAG Knows When to Stay Silent.

Most enterprise retrieval-augmented generation (RAG) demos answer questions, but very few understand that sometimes it’s better to say “I don’t know.” Here’s how we built a RAG system that refuses to guess and never leaks what it shouldn't.

Recently, our team at Momentuum blue built a RAG system that answers questions using internal documents for a fictitious company, Doofenshmirtz Evil Incorporated. These versioned documents include policies, FAQs, incident reports, and meeting minutes in PDF, DOCX, and Markdown. Every answer cites the specific policy, version and section it draws from.

The result is an enterprise RAG system that delivers trustworthy answers while reducing the operational and compliance risks associated with generative AI deployments.

How it works

• Ingestion with five configurable chunking strategies, incremental re-embedding, and version tracking that ensures newer documents supersede older ones. It’s all run on locally hosted models via Ollama with interchangeable vector stores (Chroma or Pinecone).
• Query routing that detects whether a question is a simple lookup or a comparison across policy versions, with each handled differently.
• Query rewriting that expands each question into multiple search queries, with an automatic retry using a reworded query when results are weak.
• Two-stage Matryoshka retrieval: a fast first pass with shortened embedding vectors, then a precise pass with full-length vectors.
• Hybrid retrieval combining dense and keyword (BM25) search, cross-encoder reranking and MMR (maximal marginal relevance) reranking, so results are both relevant, non-redundant and diverse.
• Context reordering to counter the "lost in the middle" effect, placing the most relevant passages where the model is most likely to use them.

Why this RAG system is built with enterprises in mind

• Access control is enforced in the retrieval layer rather than the prompt, so restricted documents never reach the model without authorization.
• When retrieval confidence is low, the system states that the answer isn't available instead of guessing.
• Every code change runs automated tests and an evaluation suite that fails the build on any restricted-document leak. On a 36-question internal evaluation set, it retrieved the correct source for every question with zero leaks.
• Each run reports latency, token usage and cost per step.
• Semantic caching reuses answers for near-identical questions to reduce latency and cost, with separate caches for each access level so restricted answers are never served to unauthorized users.

This is the kind of engineering judgment that Momentuum blue teams bring to our work for clients: systems that are accurate, auditable and secure by design.

Explore the project: github.com/nachiketht/doofenshmirtz-evil-incorporated-rag