Retrieval-augmented generation (RAG) is the most reliable way to make an LLM answer questions about your data. But a naive RAG demo and a production-grade pipeline are very different systems. The gap is almost entirely about grounding, guardrails and evaluation.
How should you chunk your documents?
Chunking decides retrieval quality more than any embedding model choice. Fixed 512-character windows shred tables, step-by-step procedures and definitions across boundaries. Chunk by document structure instead: one section per chunk with its heading attached as context ('Installation > Prerequisites'), overlap adjacent chunks by a sentence or two so answers spanning boundaries still retrieve cleanly. Store parent-section ids alongside embeddings so you can return the full section when a small chunk wins the ranking.
Plan for churn from day one: when source documents change, their embeddings must too. Keep a content-hash per chunk and re-embed only what changed — full-collection re-embeds get expensive fast, and stale vectors silently poison answers weeks before anyone notices. Budget an owner for this pipeline; 'who updates the embeddings' is the question teams forget to answer until search quality quietly rots.
Why does retrieval quality decide everything?
If the right chunk never makes it into the context window, no amount of prompt engineering will save the answer. Most 'hallucinations' in RAG are actually retrieval failures in disguise.
- Chunk semantically (by heading/section), not by a fixed character count.
- Store rich metadata alongside each chunk — source, date, permissions — and filter on it before ranking.
- Combine dense vector search with keyword/BM25 for hybrid retrieval; it dramatically improves recall on names and codes.
- Add a reranking step to push the most relevant chunks to the top of a small context budget.
How do you force the model to ground its answer?
Instruct the model to answer only from the provided context and to say it doesn't know otherwise. Then make it cite the chunk it used so the claim is verifiable.
const system = `You are a support assistant.
Answer ONLY using the <context> below.
If the answer is not in the context, say you don't know.
Cite sources as [n].`;
const messages = [
{ role: 'system', content: system },
{ role: 'user', content: `<context>${chunks}</context>\n\nQ: ${question}` },
];How do you measure RAG quality before launch?
Build an evaluation set of real questions with expected answers before you launch. Score every change against it for faithfulness (is the answer supported by the context?) and relevance. An LLM-as-judge works well here, calibrated against a few dozen human ratings.
Treat your prompt and retrieval config like code: version it, test it against a fixed eval set, and never ship a change you haven't scored.
Which guardrails belong in every production RAG stack?
- A relevance gate: if the top retrieved chunks score below a similarity threshold, answer "I don't know" instead of guessing.
- Citation enforcement: reject answers whose claims can't be mapped to a retrieved chunk id — unverified sentences get regenerated or dropped.
- Permission-aware retrieval: filter chunks by the asking user's access rights before ranking, so private documents never leak into answers.
- Prompt-injection screening: treat retrieved text as untrusted input and delimit it clearly so embedded instructions aren't followed.
- Latency and cost budgets: cache embeddings, cap context size, and log token spend per query — silent cost blowups kill RAG projects.
What does a realistic RAG eval workflow look like?
Collect 50–100 real questions from support tickets or search logs, pair each with an expected source document. For every change — new chunker, different embedding model, tweaked reranker — run the full set offline and score faithfulness (is each claim supported by cited context?) and retrieval hit-rate (did the expected doc make the top-k?). An LLM judge approximates human ratings well once you calibrate it against 30–50 hand-labelled examples. Keep the harness in CI: a change that drops faithfulness below your threshold simply cannot merge. And log production queries weekly — real user phrasing is the fastest way to discover where retrieval still fails.
References
Which vector store fits which stage?
Don't start with a dedicated vector database. If your corpus fits under roughly a million chunks, pgvector inside your existing Postgres keeps retrieval, permissions and backups in one system — transactional consistency between documents and their embeddings comes free, and you avoid a second operational surface. Managed options (Pinecone, Qdrant Cloud, Weaviate) earn their cost later: billion-scale corpora, sub-50ms p99 at high concurrency, or hybrid search primitives out of the box. Migrating between them is mostly mechanical if you abstract retrieval behind one interface — so design that seam early.
Grounding plus guardrails plus evaluation is what turns an impressive demo into a system you can put in front of customers and trust.