Architecting High-Accuracy RAG: Chunking Strategies and Re-Ranking

1. Beyond Naive Character-Count Chunking

Splitting a document every 500 or 1000 characters is the first thing every RAG tutorial teaches, and it is also the single biggest source of retrieval failures in production systems. Fixed-size chunking has no notion of what a sentence, a table row, or a code block actually is, so it routinely slices through the middle of a definition, separates a heading from the paragraph it introduces, or splits a function signature from its body. The embedding model then has to represent an incoherent fragment, and no amount of re-ranking downstream can recover semantic meaning that was destroyed at ingestion time.

The fix is structure-aware chunking: parse the document into its real units first (headings, paragraphs, list items, table rows, code blocks) using the source format's own grammar, whether that is Markdown AST parsing, HTML DOM walking, or a PDF layout extractor that preserves reading order. Chunks are then built by greedily packing these units up to a target token budget, never splitting a unit in the middle. This alone typically improves retrieval precision more than any embedding model upgrade, because it fixes the input instead of trying to compensate for it downstream.

A second, often overlooked lever is semantic chunking, where you compute embeddings for consecutive sentences and cut a chunk boundary at the point where cosine similarity between adjacent sentences drops below a threshold, indicating a topic shift. This is more expensive than rule-based splitting because it requires an embedding call per sentence at ingestion time, but for dense, unstructured prose like legal contracts or research papers, it produces noticeably more coherent chunks than any fixed-size heuristic.

Whatever strategy you pick, always attach parent context as metadata rather than trying to make every chunk self-sufficient. A chunk should carry its document title, section heading path, and source URL as structured fields alongside the vector, so that even a narrowly-scoped fragment can be correctly attributed and expanded at query time without bloating the embedding itself.

2. Parent-Document Retrieval and Contextual Chunk Headers

Small chunks retrieve well because a tightly-scoped fragment produces a sharper embedding that matches specific queries more precisely, but small chunks generate poorly because they lack surrounding context for the language model to reason over. The parent-document pattern resolves this tension by indexing small child chunks for search while storing a pointer to a larger parent chunk (a full section, or several neighboring paragraphs) that gets fetched and passed to the LLM once a child chunk is retrieved.

In practice this means your vector store holds two tiers: a fine-grained index optimized purely for match precision, and a coarser store (often just a key-value lookup in Postgres or Redis, keyed by parent ID) optimized for context completeness. This decouples the retrieval unit from the generation unit, which is the right separation of concerns and avoids the false choice between chunk sizes that hurts either precision or context.

A cheaper complementary technique, popularized as contextual retrieval, prepends a short LLM-generated summary of the chunk's place in the document (for example, this chunk describes the refund policy exception for enterprise customers, from the Q3 2024 terms of service) directly into the text that gets embedded. This costs one LLM call per chunk at ingestion time, done once and cached, and it consistently reduces retrieval failures caused by short chunks that read as ambiguous in isolation, such as a table row with numbers but no label.

3. Two-Stage Retrieval with Cross-Encoder Re-Ranking

Vector similarity search (a bi-encoder architecture) embeds the query and the document independently and compares them with a single dot product, which is what makes it fast enough to search millions of vectors in milliseconds. The cost of that speed is that the model never actually looks at the query and document together, so it misses fine-grained interactions like negation, exact term matching, or numeric comparisons that a joint representation would catch.

A cross-encoder re-ranker solves the accuracy side of that trade-off by taking the query and each candidate document as a single concatenated input and scoring them jointly through a transformer, which is far more accurate but too slow to run over an entire corpus. The standard production pattern is therefore two-stage: retrieve a wide candidate set (50 to 200 chunks) cheaply with vector or hybrid search, then re-rank only that shortlist with a cross-encoder and keep the top 5 to 10 for the LLM context.

This two-stage design is why re-ranking is close to a free accuracy win once you have retrieval working at all: the re-ranker call is a fixed, small cost (typically under 100ms for 100 candidates on a decent GPU, or a hosted API call) regardless of your corpus size, because it never touches the index itself. Teams that skip this step and try to compensate with a bigger top-k from vector search alone almost always see worse answer quality, because irrelevant-but-similar chunks dilute the LLM's context.

4. Context Window Compression and Deduplication

Once you re-rank down to the top handful of chunks, a second problem surfaces at scale: near-duplicate chunks from different document versions, or from a document that repeats the same boilerplate section, end up occupying multiple slots in the final context, crowding out genuinely distinct information. A simple maximal marginal relevance (MMR) pass after re-ranking penalizes candidates that are too similar to already-selected ones, trading a small amount of pure relevance for diversity across the final context set.

Beyond deduplication, context compression trims each selected chunk down to only the sentences that are actually relevant to the query, using either a lightweight extractive model or an LLM call with a tight prompt asking it to extract only the supporting sentences. This matters because language models exhibit a well-documented lost-in-the-middle effect, where information placed in the middle of a long context is attended to less reliably than information at the start or end, so shrinking irrelevant filler directly improves answer accuracy, not just cost.

In practice, compression is worth the extra latency and LLM call primarily for long-context use cases, such as summarizing across 20+ retrieved chunks; for typical question-answering with 5 to 8 chunks, the ordering of chunks in the prompt (most relevant first and last, per the lost-in-the-middle pattern) and simple deduplication usually deliver most of the benefit without the added cost.

5. Query Transformation and Multi-Query Retrieval

User queries are frequently underspecified, ambiguous, or phrased in ways that do not lexically or semantically match how the source documents describe the same concept. Query transformation techniques address this at the retrieval boundary rather than trying to fix it with a bigger model: HyDE (Hypothetical Document Embeddings) asks the LLM to generate a plausible answer to the query first, then embeds that hypothetical answer instead of the raw query, on the premise that an answer-shaped text embeds closer to real answer-shaped documents than a question does.

Multi-query retrieval takes the opposite approach, asking the LLM to generate three to five paraphrased or decomposed versions of the original query, running retrieval for each independently, and merging the results (typically deduplicated and re-ranked together). This is particularly effective for compound questions such as compare the pricing and support SLA of plan A versus plan B, where a single embedding of the whole question would poorly represent either sub-topic.

Both techniques add one or more LLM round-trips before retrieval even starts, so they should be applied selectively rather than on every query. A practical pattern is to run a cheap classifier or heuristic (query length, presence of comparison words, low confidence from the initial retrieval pass) to decide when the added latency of query transformation is worth it, falling back to direct retrieval for simple, well-formed queries.

6. Production Metrics & Evaluation

Shipping RAG without an evaluation harness is shipping blind, because chunking, embedding model, and re-ranker changes interact in non-obvious ways and a change that improves one query type often regresses another. The standard evaluation stack separates retrieval metrics from generation metrics: context precision and context recall (from frameworks like RAGAS) measure whether the retrieved chunks are actually relevant and whether they cover everything needed to answer, independent of what the LLM does with them afterward.

Generation-side metrics then measure faithfulness (is every claim in the answer supported by the retrieved context, which is the direct proxy for hallucination rate) and answer relevance (does the answer actually address the question asked, since a faithful answer can still dodge the question). Both are typically scored by a separate, stronger LLM acting as a judge against a labeled or synthetically-generated evaluation set of query, expected-context, and expected-answer triples.

The operational discipline that makes this sustainable is building the evaluation set incrementally from real production failures: every time a user flags a wrong answer or thumbs it down, that query and the correct expected answer get added to a regression suite that runs on every pipeline change, in the same spirit as unit tests. Without this feedback loop, most teams tune RAG once at launch and then fly blind as the document corpus and user query patterns drift over months.

7. Synthesis: A Production Checklist

High-accuracy RAG is not one trick but a pipeline of compounding fixes, each addressing a different failure mode: structure-aware chunking fixes broken semantic units at ingestion, contextual headers and parent-document retrieval fix loss of context, hybrid search and re-ranking fix precision at retrieval time, and deduplication plus compression fix context-window dilution at generation time. Skipping any one stage does not make the pipeline fail outright, it just caps the achievable accuracy ceiling below what the business actually needs.

If you are starting from a naive fixed-size-chunk, top-k-vector-search baseline, the highest-leverage next steps in order are: switch to structure-aware chunking, add a cross-encoder re-ranker over a wider candidate set, and build even a small (50 to 100 query) evaluation set before touching anything else. These three changes alone typically account for the majority of the achievable accuracy gain, and they are all measurable in isolation, which matters because you want to know which change actually moved the needle.

Finally, treat the evaluation set as a living asset that grows with every production failure, not a one-time launch artifact. RAG accuracy is not a property you achieve once; it is a property you maintain against a document corpus and a user base that both keep changing, and the only durable defense against silent regressions is a regression suite that runs before every deploy.