Almost every RAG demo we’ve seen in the last two years looks magical, and almost every RAG system in production we’ve been asked to fix in the last twelve months has had the same set of problems. The gap between demo and production is not the model — it’s everything around the model.
When we ship a retrieval-augmented generation system, the first thing we build is the eval harness. Not the prompt. Not the embedding pipeline. The harness — usually 200–2,000 ground-truth examples that capture the questions our system has to answer, with the answers we know are correct.
Without that harness, every change to the system is a guess. With it, every change is measured. Six months in, when the model gets upgraded, you have a one-click answer to whether the upgrade helped or hurt.
The second thing we build is hybrid retrieval. BM25 catches what vector search misses. Re-ranking catches what hybrid retrieval misses. Query rewriting catches what bad user questions miss. Each of those layers buys you measurable recall, and you should know which one is doing the work.
The third thing — and this is the part most teams skip — is observability. Every retrieval, every prompt, every response, logged with the trace ID that ties it back to the original user question. When something goes wrong six months from now, you want to be able to answer ‘why did it answer this?’ in less than a minute.
RAG is one of the most useful things you can build right now. It’s also one of the easiest to ship a demo of and the hardest to keep alive in production. Don’t skip the unglamorous parts.
Written by the brainiac/studio team. We publish original work from the engineers, designers, and marketers who do the work — never outsourced to a content shop.