What is chunking? The first critical decision for RAG quality

Chunking is splitting a long document into small, searchable segments before you build a RAG retrieval system. Each segment gets converted to a vector and stored in a database. When a user asks a question, the system retrieves the most relevant segments and feeds them to the language model. Chunking determines what the model can see, and the model can only answer based on what it sees.

When RAG retrieval underperforms, the problem is usually not the model itself, but the chunking step that precedes it.

What is chunking? Why it's your first critical decision

Say you have a thirty-page supply contract and want to turn it into an internal Q&A tool. You can't feed all thirty pages to the model with every question. The context window won't fit, and even if it does, there's too much noise for the model to find the relevant information. You need to split it up first.

The question is how to split it. There's no standard answer, and different chunking approaches will give you completely different results for the same question.

Here's an example. A contract might contain a passage like this:

Article 8: Payment Terms. The buyer shall pay the full invoice amount within thirty days of acceptance. For late payment, penalties accrue daily at 0.05% of the outstanding balance. Article 9: Warranty. The seller provides a twelve-month warranty from delivery...

Now a user asks: "How are late payment penalties calculated?"

The same contract, three chunking methods, three different answers

Method one is fixed-length hard chunking. This approach ignores meaning and cuts every 200 characters. The phrase "penalties accrue daily at 0.05% of the outstanding balance" gets cut between "penalties accrue daily at" and "0.05% of the outstanding balance", with parts appearing in two chunks. When the system retrieves the first chunk, the model sees "penalties accrue daily at" without knowing the rate. It either gives a vague answer or invents a number. The root cause isn't a weak model, but chunking that breaks apart the answer.

Method two: semantic boundary chunking. Cut along natural boundaries such as article numbers and periods. "Article 8: Payment Terms" stays intact. The system retrieves the complete payment clause, and the model correctly answers "0.05% daily." This approach usually works better than fixed-length chunking.

Method three: overlapping chunks plus context. Semantic chunking has limitations. If a user asks "Is there a relationship between the warranty in Article 9 and the payment terms in Article 8?" and each article is chunked independently with no overlap, the model won't see the connection. Adding overlap (adjacent chunks share a section) and metadata like "From Contract X, Article 8" helps. With this context, cross-clause questions are answered correctly.

The three approaches compare as follows:

No best chunking method, only the right one for your data

Using method three for everything isn't practical. Overlapping chunks increase your chunk count, which raises vector storage and retrieval costs. Chunking too small loses information density; too large brings in noise. Chunk size, overlap percentage, and whether to preserve table and heading structure all involve trade-offs. The right choice depends on your document format and what questions users will ask.

Contracts and spec sheets require different chunking approaches. Medical patient education materials and manufacturing SOPs are different again. Examining your specific data and choosing parameters accordingly works better than applying generic settings.

This lesson comes from experience: a project set chunking parameters once and never adjusted them. Three months in, users reported declining accuracy. Investigation revealed the document format had changed, adding tables, and the original chunking was breaking them apart. Chunking requires ongoing maintenance as document formats and data change.

Treat chunking as the first engineering decision in any RAG project. Run experiments with actual documents, validate retrieval hit rates against real questions, then pick a strategy. Don't deploy the model and fix chunking problems later. An impressive demo matters less than reliable performance months after launch.