The production RAG checklist: 23 things before you launch

Most RAG demos die on fresh data, permissions, and cost. Here's the accelerated-infra version we use for every launch:

Retrieval (the 80%)

• Hybrid dense + BM25, not pure vector
• Reranker (bge / Cohere) on top-50 → top-6
• Chunk by structure, 400–800 tokens with overlap
• Permission filtering at retrieval, not post-hoc
• Nightly re-index + deleted-doc tombstones

Generation (the 15%)

• Citations enforced, “I don’t know” allowed
• Prompt caching + semantic cache (35–55% hit)
• Distilled 8B for 80% of queries, 70B fallback

Evals & Ops (the 5% that saves you)

• 200+ golden Q/A, regression CI blocks deploys
• TTFT / TPOT / $/1K queries dashboard
• PII redaction + jailbreak suite

vLLM --model distilled-8B-awq \
  --enable-prefix-caching \
  --max-num-seqs 256 --gpu-memory-utilization 0.92

We build this in 4 weeks →