Why Your RAG System Is Actually a Search Problem

I’ve reviewed a fair number of RAG systems that “mostly work,” and the complaint is almost always the same. The demo went beautifully. Then real questions started coming in, and the answers turned vague, or confidently wrong, or quietly skipped the one document that actually held the answer. The first instinct, every time, is to reach for a better model.

That instinct is usually wrong. People blame generation: swap the model, raise the temperature, rewrite the prompt, pile on few-shot examples. Sometimes it helps a little. Most of the time it doesn’t, because the model was never the bottleneck. A language model can only reason over what you hand it. If you hand it the wrong chunks, it writes a fluent answer to the wrong question.

Here’s the point I keep coming back to: a RAG system is a search engine with a language model bolted on the end. Get retrieval right and almost any competent model produces a good answer. Get retrieval wrong and no model saves you. So most of what we call “RAG quality” work is really information retrieval, a field that’s been refining these exact problems for about fifty years. That doesn’t mean generation never matters. It means you can’t honestly diagnose generation until retrieval is good, and retrieval is where the problem usually lives.

A RAG system is a search engine with a language model bolted on the end. If retrieval surfaces the right context, the generation was never the hard part.

 Read Now : How much Postgres handles on its own ?

Why do good RAG demos fall apart on real questions?

Because demos use questions whose answers are easy to find, and real users don’t. When you build the demo, you ask questions you already know the corpus can answer, often phrased close to how the source text phrases it. Retrieval looks flawless. Then production hits it with the long tail: questions worded nothing like the document, questions that span three sections, questions whose answer is one sentence buried in a forty-page PDF.

The retriever that aced your softball questions has no special talent for the hard ones. It was never tested on them. This is the oldest lesson in search: the average query is easy and the tail is brutal, and the tail is where users actually live. If you only ever evaluate on questions you wrote yourself, you’re measuring the wrong distribution.

Isn’t vector similarity the same as relevance?

No, and the gap between the two is where most RAG systems leak. Vector similarity finds text that sits near your query in embedding space. Relevance is text that answers your query. Those overlap often enough to be seductive, but they are not the same thing.

Take a support question like “how do I cancel my plan.” Embeddings will cheerfully return chunks about plans, pricing tiers, and upgrades, all genuinely “similar,” none of which tell the user how to cancel. Embeddings also stumble on the literal: error codes, function names, SKUs, version strings, the exact token someone typed. Cosine similarity has no particular respect for ERR_4032. A keyword index does.

Where’s the line? Dense vectors are genuinely good at the thing keyword search is bad at: paraphrase, synonyms, conceptual matching. The failure isn’t vectors. It’s vectors alone, asked to carry a job that the IR field solved long ago by combining methods.

Read Now : Why most teams reach for a vector database too early ?

Should you put keyword search back into a vector pipeline?

Usually yes. Hybrid search, dense vectors plus old-fashioned lexical search, beats either approach by itself on most real corpora, because the two fail in opposite directions. BM25-style lexical retrieval nails exact terms, rare words, identifiers, and names. Dense retrieval nails meaning when the words don’t match. Run both, then fuse the ranked lists. Reciprocal rank fusion is the simple default that holds up well and needs no tuning.

You don’t need a second system for this. Postgres does both in one query: pgvector for the dense side, the built-in full-text search for the lexical side, combined and fused in SQL.

-- Dense plus lexical, fused with reciprocal rank fusion.
-- Over-retrieve from each side, then combine the rankings.
WITH dense AS (
  SELECT id,
         1.0 / (60 + ROW_NUMBER() OVER (ORDER BY embedding <=> :query_vec)) AS score
  FROM chunks
  ORDER BY embedding <=> :query_vec
  LIMIT 100
),
lexical AS (
  SELECT id,
         1.0 / (60 + ROW_NUMBER() OVER (ORDER BY ts_rank(tsv, q) DESC)) AS score
  FROM chunks, plainto_tsquery('english', :query_text) q
  WHERE tsv @@ q
  LIMIT 100
)
SELECT id, SUM(score) AS rrf_score
FROM (SELECT * FROM dense UNION ALL SELECT * FROM lexical) fused
GROUP BY id
ORDER BY rrf_score DESC
LIMIT 20;

Where’s the line? If your queries are pure conceptual paraphrase and never contain a literal token that matters, dense alone might be enough. That’s rarer than it sounds. Almost every real domain has identifiers, names, or codes hiding in the questions that users actually ask.

Why does reranking matter more than a better embedding model?

Because reranking fixes precision, and precision is usually what’s broken. Retrieval is a two-stage game: first recall, then precision. Pull a generous fifty to a hundred candidates with fast retrieval so the right chunk is somewhere in the set. Then rerank that set with a cross-encoder, a model that reads the query and each chunk together and scores how well they actually match, and reorder by that score. The five chunks you finally hand the language model get dramatically better.

In my experience this is the best cheap win in the whole pipeline. A reranker is often a larger quality jump than upgrading your embedding model, costs less, and needs no re-indexing. The bi-encoder embeddings you search with are fast but shallow, because the query and the document never meet until you compare two pre-computed vectors. A cross-encoder lets them meet. That’s where the precision comes from.

Where’s the line? Reranking adds a model call and some latency. If you only retrieve five chunks and feed all five, there’s nothing to rerank. It earns its place when you can over-retrieve and you have more good candidates than context budget, which is the normal situation once a corpus gets real.

Is chunking part of the model or part of the search?

Chunking is search, and it sets the ceiling on everything above it. How you split documents decides what can ever be retrieved as a clean unit. Cut a corpus into fixed five-hundred-token windows that slice through the middle of a procedure, and no retriever on earth can return that procedure intact, because it doesn’t exist as a retrievable unit. You destroyed it at index time.

Split on structure instead: sections, headings, natural semantic boundaries. Keep each chunk self-contained enough to stand alone. Carry a little overlap so context doesn’t fall off a cliff at the edges. Attach metadata like source and section title so you can filter and so the model knows where the text came from. None of this is new. Search engineers have always known the index design determines the ceiling, and chunking is index design wearing a new hat.

Where’s the line? There’s no universal chunk size. Dense reference docs, long narratives, and source code each want a different strategy, and you find it by looking, not by guessing. The real line is simpler: if you’ve never read what your chunks actually contain, you’re tuning the model blind.

So how do you tell a retrieval problem from a generation problem?

You measure them separately, in order. This is the part most teams skip, and it’s the part that saves the weeks.

1. Take a set of real questions the system gets wrong. Pull the chunks it retrieved for each, and read them. Is the answer actually in there? If it isn’t, you have a retrieval problem, full stop. No prompt, no model upgrade, no temperature setting fixes a document that never showed up.
2. If the answer is present in the retrieved chunks and the model still gets it wrong, now you have a generation problem worth working on: prompt, context ordering, or model choice.
3. Track retrieval as its own metric, not just end-to-end vibes. Recall at k tells you whether the right chunk made the top k at all. Rank position tells you whether it survived the cut into the context you actually sent.
4. Fix retrieval first. The order is not optional, because retrieval failures disguise themselves as generation failures, and if you start at the model you’ll tune the wrong layer for a month.

So why do teams keep blaming the model? Because the model is the visible, exciting part, and the retriever is plumbing. The plumbing is where the answers are.

 Read Now : The real cost of an LLM call vs a SQL query 

What to build instead

Build the search engine first. Hybrid retrieval with dense and lexical side by side, over-retrieve a generous candidate set, rerank with a cross-encoder, hand the model the top few, and only then spend time on the prompt and the model. Chunk on structure, not on token counts. Evaluate retrieval on its own, with real questions and recall numbers, before you blame anything downstream.

The cautionary version is easy to picture, because it’s everywhere. A team ships RAG, the answers are mediocre, and they spend the next quarter swapping models and rewriting prompts while the retriever quietly returns the wrong paragraph every time. Nobody measured recall. The model was never the problem.

A RAG system is a search engine with a language model on the end. Treat the search like it matters, because it’s the part that decides whether any of the rest is worth running.