Chunking

How it works

A 60-page policy manual is too big to retrieve as one unit, and a single embedding cannot represent 60 pages of meaning. So the document is cut into chunks, and each chunk is embedded and stored with metadata: source file, page, section heading, date.

Common approaches:

• Fixed size: every 300 to 800 tokens, with 10% to 20% overlap so sentences at the edges are not lost
• Structure-aware: split on headings, list items and table boundaries, keeping each section whole
• Semantic: split where the topic changes, measured by embedding similarity between sentences
• Parent and child: search small chunks for precision, then hand the model the larger section around the match

Structure-aware chunking with a size cap is our usual starting point. It keeps a clause or a procedure in one piece.

How businesses use it

Chunking is a design step inside every RAG system, and it decides more about answer quality than the choice of model. It matters most for:

• Contracts, where a clause only makes sense with its definitions (see the contract review assistant blueprint)
• Technical manuals with numbered procedures
• Spreadsheets and tables, which need their header row repeated in every chunk
• Scanned PDFs, which need document processing to extract clean text first

Where it goes wrong

Bad chunking produces answers that sound right and are not. A table split across 2 chunks loses its headers. A procedure cut in the middle returns step 4 without steps 1 to 3. A chunk that says "this does not apply to contractors" is useless once it has lost the heading that says what "this" is.

Start by adding the document title and section heading to each chunk's text before embedding. Anthropic's contextual retrieval tests (opens in a new tab) (September 2024) found that adding a short explanation of context to each chunk cut retrieval failures by 35%. Then test retrieval on real questions and read the passages that come back, not only the final answers. If exact terms still go missing, add hybrid search.