What is TokenTextSplitter and why is document chunking necessary?

Before documents can be embedded and stored in a VectorStore, they must be split into smaller pieces called chunks. TokenTextSplitter is Spring AI's built-in chunking utility that divides large documents into token-bounded segments while trying to preserve sentence and paragraph boundaries.

Chunking is necessary for two reasons. First, embedding models have an input token limit (typically 512–8192 tokens). A 50-page PDF would exceed any model's limit, so it must be split before embedding. Second, retrieval quality improves with smaller, focused chunks — returning a 200-token paragraph precisely about your question is far more useful than returning a 5000-token document that might contain the answer buried inside unrelated text.

TokenTextSplitter splitter = new TokenTextSplitter(
    600,    // target chunk size in tokens
    100,    // overlap — tokens shared between adjacent chunks
    5,      // minimum chunk size
    10000,  // max chars per chunk (safety cap)
    true    // keep separators
);
List<Document> chunks = splitter.apply(rawDocuments);

The overlap parameter is important: it makes adjacent chunks share a window of tokens. This prevents relevant context from being split exactly at a chunk boundary, so a sentence that straddles two chunks can still be found during retrieval.

Alternative splitters include CharacterTextSplitter (splits on character count) and you can implement TextSplitter directly for custom logic — for example, splitting Markdown documents at heading boundaries.