What are document loaders and splitters?

Document loaders ingest content from various sources and return a list of Document objects (each containing page_content and metadata). Text splitters then divide those documents into smaller chunks suitable for embedding and retrieval.

Common document loaders:

• PyPDFLoader — extracts text from PDF files, one page per Document
• WebBaseLoader — scrapes a web page, returns its text content
• CSVLoader — each row becomes a Document
• DirectoryLoader — recursively loads all files in a directory
• UnstructuredFileLoader — handles Word, PowerPoint, HTML, email, and more
• GitHubLoader — loads files from a GitHub repository

Common text splitters:

• RecursiveCharacterTextSplitter — splits on paragraphs, then sentences, then words until chunks fit the target size. Most commonly used.
• CharacterTextSplitter — splits on a single character separator
• TokenTextSplitter — splits by token count, precise for context window budgeting
• MarkdownHeaderTextSplitter — splits Markdown by header sections, preserving structure

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,    # max chars per chunk
    chunk_overlap=200,  # overlap to preserve context at boundaries
    length_function=len,
)
chunks = splitter.split_documents(documents)