RAG chunking: four strategies and parameters

We helped an insurance company optimize RAG last week. They had loaded three thousand policy documents into their knowledge base, but Q&A accuracy was stuck at 60%. The engineering team wanted to try swapping models or embeddings first. We investigated and found that wasn't the issue. The problem was chunking.

A 20-page policy document had been cut into 512-token chunks. The definition of 'waiting period' and its exceptions ended up in separate chunks. During retrieval, the system retrieved only half the information. The model saw incomplete context and gave scattered answers. We changed the chunking strategy. Same model, same documents. Accuracy reached 85% that day.

This article explains how RAG chunking works, where the four main strategies differ, and what starting chunk size and overlap values to use for each.

Start with a solid definition

RAG chunking is the process of splitting raw documents into individual text blocks (chunks) that are semantically coherent and sized appropriately for vector retrieval when building a retrieval-augmented generation system. Chunking determines your retrieval's minimum unit of meaning: chunk too large and one block contains too many topics, hurting retrieval precision; chunk too small and individual blocks lose context, leaving the model without enough information. Chunk quality directly sets the ceiling on your RAG system's retrieval quality.

Your embedding model and LLM are off-the-shelf. Chunking is the lever you can actually tune. It's the cheapest and delivers the highest return.

Four chunking strategies

Fixed-size chunking. Count tokens and cut at 512, with overlap to make adjacent chunks overlap slightly. Fast and predictable. You can implement it in ten lines of code. The problem is that it completely ignores sentence and paragraph boundaries, often splitting complete ideas in half. Best for loosely structured, uniform-content long documents such as interview transcripts or internal chat logs.

Recursive chunking. This is most teams' default starting point. LangChain's RecursiveCharacterTextSplitter is built on this. The approach uses a hierarchy of separators: first try splitting on paragraph breaks, and if chunks are still too large, fall back to sentence breaks, and if still not enough, drop to character-level splits. It cuts along natural boundaries while guaranteeing you stay under your size limit. Between quality and engineering cost, it usually offers the best price-to-performance ratio.

Semantic chunking. Ignore length, look at meaning instead. Split the document into sentences first, then calculate embeddings for each sentence. When vector similarity between adjacent sentences drops sharply, mark that as a topic shift and cut there. Each chunk is thematically clean and boundaries feel natural. The cost is an extra embedding pass during indexing, making it slower and more expensive. Worth it for documents with high topic drift and strict retrieval precision demands, like regulations or research reports.

Document-structure chunking. Cut along the document's own framework: follow heading levels in Markdown, follow the DOM in HTML, follow chapters in PDF, or follow functions in code. The prerequisite is that the document has clear structure. Our insurance client, for example, had policies with explicit section, subsection, and clause numbering. When we chunked by structure, each chunk aligned perfectly with a complete clause. That's what actually moved the needle on their accuracy.

Quick reference: starting parameters

Below are starting values from real projects. These aren't theoretical optima; they're anchors to set first, then fine-tune up or down from. Token counts assume Chinese-heavy content; for English, you can roughly multiply by 1.5.

A few important considerations. Bigger overlap isn't always better. Push it above 30% and your index expands, and at retrieval time duplicate chunks crowd out the truly relevant ones, which reduces your accuracy. Chunk size also needs to work with your embedding model's input limit and the context budget left for the LLM. Chunk too large and your top-k retrieval of three chunks exceeds your context window. Don't forget to attach metadata like 'source document' and 'section heading' to the front of each chunk. That lets you filter during retrieval and cite sources in your answers.

The best strategy is the one that fits your data

Consider this principle: look at whether your documents have structure first. If they have clear headings, numbering, and sections, lead with document-structure chunking and use recursive as your fallback. For loosely structured content, start with recursive. If precision falls short, add semantic chunking for the problem areas. Reserve fixed-size for when you really just need to run a POC fast.

Chunking isn't a set-it-and-forget-it decision. Once you're live, monitor your retrieval logs: which queries are pulling back semantically incomplete chunks? Which topics consistently underperform? Go back and tune your size and overlap. It's an iterative process.

When we roll out RAG at Tenten, we rarely try to engineer perfect chunking from the start. Usually we run a baseline with recursive chunking first, then take the customer's real question set and measure retrieval hit rates, and only then swap in structure-based or semantic chunking for the document categories that underperformed. Perfect chunking in a demo doesn't count. What matters is the accuracy you measure in the customer's environment with real query volume. That's the number we go by.