Before you change the model, look at your chunks

The chunk is the basic unit of evidence a retrieval system works with. If that unit is incoherent, retrieval quality drops without any warning. Good chunking makes the rest of a retrieval-augmented generation (RAG) pipeline easier; bad chunking forces every later component to compensate for broken evidence.

So before you swap models, before you upgrade the vector database, and certainly before you start building multi-agent loops, open the index and read a few chunks. Check that they actually mean something.

Why it fails quietly

What makes chunking dangerous is that it never throws an error. A badly split document still indexes, the search still returns ten results, and the model still answers, fluently and with confidence. Nothing in the pipeline reports a problem. The answer is just wrong, or vague, or built on half a table.

The usual reaction is to blame the model and try a bigger one. The bigger model then writes a more eloquent version of the same wrong answer, because it was handed the same broken evidence.

A typical case on a plant: a maintenance technician asks the assistant for the torque setting of a pump coupling. The assistant answers with a value from the right manual, but from the table for the other pump size, because the table was split and the size column ended up in another chunk. Nobody notices until someone checks the paper manual.

What breaks first on industrial documents

Technical documentation punishes naive splitting much harder than prose does. In our field the same failures come back again and again.

• Tables cut in half. A parameter table split between rows leaves the header in one chunk and the values in another, and retrieval returns numbers with no idea what they describe.
• Procedures separated from their conditions. Step 7 retrieved without the safety precondition stated in step 1 is worse than no answer.
• Orphaned references. “As described in section 4.2” means nothing when section 4.2 sits in a chunk the model never sees.
• Units and ranges cut off from their subject. A chunk that contains “4–20 mA” and nothing else tells the model nothing useful.
• Lost headers. Strip the document title and the section path, and a chunk about “the controller” no longer says which controller.

What we do instead

There is no universal chunk size, but there is a method.

Split on structure before splitting on length. Headings, numbered sections and table boundaries carry meaning that a character count does not. Then keep a breadcrumb in every chunk: document title, section path and revision. It costs a few tokens and saves the model from guessing which manual, and which version of it, it is reading.

Use overlap on purpose. Overlap repairs boundaries you could not place well; where the structure gives a clean boundary, it only adds noise. Keep tables whole, and if a table is too large to fit, convert it to sentences rather than cutting it.

Finally, test with real questions. Write down twenty questions your colleagues actually ask, and check whether the retrieved chunks contain the answer. That simple measurement is worth more than any amount of tuning by intuition.

Retrieval quality is decided at ingestion, long before anyone asks a question.

Sources

• The article that prompted this: RAG Chunking That Works: Semantic Splitting, Overlap, and Eval-Driven Tuning.
• Also shared as a LinkedIn post.