Chunking

Chunking means: a long text is cut into smaller pieces. These pieces are called chunks, roughly translating to bits or sections. A 300-page manual, for example, becomes a few thousand paragraphs of just a few sentences each. This is necessary because programs designed to understand and answer questions about texts can only process a limited amount of text at once. Instead of feeding the entire manual to the system, one retrieves the three or four sections that match the question. Cutting up text sounds trivial, but it often determines whether a system delivers useful or useless answers.

Why the cutting points determine answer quality

Most companies don't build their AI applications by training their own model. They take an existing model and give it access to their own documents. For this to work, the system must find the right spot among thousands of pages. The search is never conducted across entire documents, but always across chunks.

If the chunks are too large, each match comes with a lot of extra baggage. The model then receives, alongside the information sought, three unrelated topics as well, and easily confuses them. If the chunks are too small, context gets lost. A section that only states "The deadline is 14 days" is worthless if nowhere does it say which deadline is meant.

A common mistake is cutting in the middle of a table or in the middle of a sentence. Such torn chunks lead to answers that seem plausible at first glance but contain incorrect figures. Anyone puzzled by poor results from a document search will astonishingly often find the cause not in the model, but in the way the text was cut up.

From fixed character grids to content-based separation

The simplest method counts characters. For example, a new piece is cut off every 1000 characters. This is fast and completely independent of content, but ignores all structure. It's somewhat better to split at natural boundaries: at paragraphs, headings, or sentence ends. For a text with clear structure, this usually delivers good results.

Almost always, overlap is used in addition. This means: the end of one chunk is repeated at the beginning of the next, for instance the last two sentences. This way, no information gets lost that happened to sit exactly on the cutting edge. You can picture it like roof tiles overlapping each other so no rain gets through the gaps.

More elaborate methods cut according to meaning. A program checks where the topic in the text changes and sets the boundary there. This costs more computing time but adapts to unstructured texts. For program code, tables, or contracts, there are again separate rules, because different units make sense there: a function, a line, a paragraph.

Chunking in corporate chatbots and search systems

You most often encounter chunking in systems that retrieve answers from a company's own document collection. This approach is called Retrieval Augmented Generation, or RAG for short: the system first searches for matching passages and then lets the model answer based on this foundation. Every internal corporate chatbot, every AI-powered help page, and many research assistants work this way. Without chunking, there would be nothing to search through.

In product descriptions and technical articles, the term usually appears together with a number, such as "chunk size 512 tokens." A token here is a unit of text roughly half a syllable to a full syllable long. 512 tokens correspond roughly to half a page of text. Such figures are adjustment knobs that developers turn when fine-tuning.

A common misconception is that larger context windows make chunking unnecessary. Modern models can indeed read very long texts at once. Nevertheless, cutting text up remains sensible, because you don't want to send the entire documentation along with every single question. That would be expensive, slow, and would let the relevant passage get lost in the noise.