What is Retrieval-Augmented Generation (RAG) in LangChain4j and how do you build a pipeline?

RAG (Retrieval-Augmented Generation) is the technique of enriching an LLM prompt with relevant external content retrieved from a knowledge base before asking the model to generate a response. It solves the core limitation of LLMs — their knowledge is frozen at training time — by dynamically injecting up-to-date or domain-specific content at inference time.

In LangChain4j, a RAG pipeline has two distinct phases:

Ingestion phase (run once or periodically): Load documents → split into chunks → embed each chunk → store vectors in an EmbeddingStore.

Retrieval phase (at query time): Embed the user query → similarity-search the EmbeddingStore → inject top-K relevant chunks into the prompt → call the LLM.

// --- Ingestion ---
EmbeddingModel embeddingModel = new OpenAiEmbeddingModel.Builder()
    .apiKey(apiKey).modelName("text-embedding-ada-002").build();

EmbeddingStore<TextSegment> store = new InMemoryEmbeddingStore<>();

List<Document> docs = FileSystemDocumentLoader.loadDocuments("./docs");
EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder()
    .documentSplitter(DocumentSplitters.recursive(500, 50))
    .embeddingModel(embeddingModel)
    .embeddingStore(store)
    .build();
ingestor.ingest(docs);

// --- Retrieval at query time via AI Services ---
interface Assistant {
    String answer(String question);
}

Assistant assistant = AiServices.builder(Assistant.class)
    .chatLanguageModel(chatModel)
    .contentRetriever(EmbeddingStoreContentRetriever.from(store))
    .build();

String answer = assistant.answer("What are our refund policies?");

LangChain4j also supports advanced RAG patterns like query compression, re-ranking with a cross-encoder, and multiple content retrievers that are combined via a DefaultRetrievalAugmentor. These address quality issues in naive RAG implementations where retrieved chunks are too generic or poorly ranked.