# chunking Also called: text splitting, document splitting, passage segmentation Chunking is the preprocessing step where you split a source document into smaller, self-contained pieces before you embed them, index them, or feed them to a model. Each chunk becomes the unit your retriever returns and your model reads, so its size and boundaries decide whether a query finds the passage that answers it. Chunk too small and each piece loses the surrounding context that made it useful; chunk too large and one irrelevant sentence buries the paragraph you actually wanted. ## What a chunk actually is A chunk is a contiguous slice of a source document, produced by a splitter and stored as the unit your retriever indexes. In a RAG pipeline the chunk is what gets embedded, what gets returned by a similarity search, and what the model reads at answer time. If your splitter cuts a sentence in half, that half sentence is what your embedding model sees; if it groups an entire chapter into one chunk, that chapter is what your context window has to swallow. Chunking sits between loading and embedding. Frameworks make the boundary explicit: LangChain calls the objects that do it text splitters and defines their job as breaking large documents into pieces that will be retrievable individually and fit inside the model's context window. LlamaIndex calls them node parsers and says a parser takes a list of documents and chunks them into node objects, each node a specific slice of its parent. Same job, different vocabulary. ## How big should a chunk be? Two hard limits set the ceiling. The embedding model has a token cap, for example 8,191 tokens for OpenAI's text-embedding-3-small, and anything longer is silently truncated or rejected. The generation model has a context window, and every chunk you retrieve is competing with every other chunk plus the prompt and the reply for that budget. If you retrieve five chunks and your chunk size is 4,000 tokens, you have already spent 20,000 tokens of context before the model reads the question. The floor is set by meaning. A chunk of 50 tokens is unlikely to contain a complete idea, so it embeds as a vague fragment and retrieves for the wrong queries. In practice most RAG stacks land between roughly 200 and 800 tokens per chunk with a small overlap of 10 to 20 percent to stop ideas from being cut at the boundary. Those numbers are conventions, not laws: the right size depends on how self-contained your source paragraphs already are and how narrowly your users ask questions. ## Splitting strategies that survive real documents The naive approach is fixed-size character or token splitting, which is cheap and reproducible but cuts across sentences and code blocks. Recursive splitters improve on this by trying a list of separators in order, for example paragraph, then newline, then sentence, then word, and only falling back to a raw character cut when nothing else fits. LangChain's RecursiveCharacterTextSplitter is the default recommendation for prose for exactly this reason: it keeps paragraphs intact when it can. Structure-aware splitters do better again when the source has structure to lean on. Markdown, HTML and code splitters cut on headings, tags or syntax nodes, so a function does not get sliced across its signature and a section stays with its heading. Semantic chunkers go one step further and use embedding distance between adjacent sentences to guess where the topic changes. And Anthropic's contextual retrieval writeup shows a different angle entirely: keep the chunks small, but prepend a model-generated sentence of surrounding context to each one before embedding, which cut their retrieval failure rate substantially. ## The mistakes that quietly ruin retrieval The first mistake is treating chunk size as a global constant. A 512-token window that works for policy prose will shred a 20-line code sample and swallow a two-paragraph FAQ answer whole. Splitters that respect document structure avoid most of this, but only if you actually feed a Markdown splitter Markdown, not the plain text export of it. The second mistake is forgetting that chunks are model-specific in the same way embeddings are. Change the chunk size, the overlap, or the splitter, and every stored vector was produced under different rules than the query that will hit it. That does not always break search, but it degrades it quietly, and the only clean fix is to re-embed. Store the splitter name, chunk size and overlap alongside each vector so a future migration can tell current chunks from stale ones. The third mistake is skipping evaluation. Chunking has no single right answer, so the only way to know your settings help is to run a fixed set of queries against a fixed set of documents and measure whether the right passage shows up in the top results. Everything else is folklore. ## Recursive character splitting in LangChain from langchain_text_splitters import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter( chunk_size=500, # target size in characters chunk_overlap=75, # ~15% overlap between neighbours separators=["\n\n", "\n", ". ", " ", ""], length_function=len, ) doc = open("handbook.md").read() chunks = splitter.split_text(doc) for i, c in enumerate(chunks[:3]): print(f"--- chunk {i} ({len(c)} chars) ---") print(c) # Persist with the settings that produced it, so a future # re-index can tell current chunks from stale ones. metadata = { "splitter": "RecursiveCharacterTextSplitter", "chunk_size": 500, "chunk_overlap": 75, "source": "handbook.md", } The separators list is tried in order: the splitter first tries to break on blank lines, then single newlines, then sentence ends, and only falls back to raw characters when nothing else fits. Storing the splitter settings with each chunk is what makes a later migration debuggable. ## Frequently asked Q: Is there a single best chunk size? A: No. The useful range for prose in most RAG stacks is roughly 200 to 800 tokens with 10 to 20 percent overlap, but the right value depends on how self-contained your source paragraphs are and how narrowly users ask questions. The only reliable way to pick a number is to run a fixed evaluation set against a few candidate settings and see which one puts the right passage in the top results. Q: Do I need overlap between chunks? A: Usually yes, but only a little. Overlap of 10 to 20 percent keeps an idea that straddles a boundary from being split cleanly in two, so both neighbouring chunks retain enough context to be retrievable. Larger overlaps waste embedding tokens and storage and let the same sentence dominate multiple retrieval hits. Structure-aware splitters that respect paragraph and heading boundaries can often use less overlap than raw character splitters. Q: What is semantic chunking and is it worth it? A: Semantic chunkers embed sentences one at a time and cut where the embedding distance between neighbours crosses a threshold, so the boundaries land at topic shifts rather than at character counts. It costs extra embedding calls at ingest time and is sensitive to that threshold. It tends to help when your sources are long and topically uneven, and to be overkill when your sources are already short, well structured articles. Q: If I change my chunking, do I have to re-embed everything? A: In practice yes. Every vector in your index was produced from a chunk shaped by a specific splitter, size and overlap, so mixing new and old vectors in the same index gives you queries that silently prefer whichever regime happens to embed closer to them. The clean fix is to re-embed the corpus under the new settings, which is also why storing splitter metadata alongside each vector is worth the extra column. ## Sources - LangChain docs: Text splitters: https://docs.langchain.com/oss/python/integrations/splitters/ defines text splitters, the recursive character splitter, and document-structure-based splitters for Markdown, HTML, JSON and code - LlamaIndex docs: Node parsers: https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/ the canonical description of node parsers as the abstraction that chunks documents into retrievable nodes - OpenAI Cookbook: Embedding texts longer than the model's maximum context length: https://developers.openai.com/cookbook/examples/embedding_long_inputs shows how the 8,191-token embedding cap forces you to either truncate or chunk, and why chunking on paragraph or sentence boundaries preserves meaning - Anthropic: Introducing Contextual Retrieval: https://www.anthropic.com/news/contextual-retrieval demonstrates prepending model-generated context to each chunk before embedding, and reports the resulting drop in retrieval failures ## Related terms - RAG: https://ai.developertools.io/rag/index.md - embedding: https://ai.developertools.io/embedding/index.md - context window: https://ai.developertools.io/context-window/index.md - token: https://ai.developertools.io/token/index.md - memory: https://ai.developertools.io/memory/index.md Canonical page: https://ai.developertools.io/chunking/ Edited by Derek Wonk. Last updated 2026-08-28.