Semantic Text Splitting With Overlap And Metadata Preservation

via “text splitting and document chunking with semantic awareness”

Framework for building LLM applications with chains, agents, retrieval, and tool use.

Unique: Provides TextSplitter abstraction with multiple strategies (character, token, semantic, language-specific) and automatic metadata propagation. RecursiveCharacterTextSplitter preserves document structure by splitting on progressively smaller delimiters, while TokenTextSplitter respects LLM token limits.

vs others: More comprehensive than simple string splitting and more flexible than rigid chunking strategies because splitters are composable and support language-specific and semantic-aware approaches.