How does LangChain4j support multi-modal input processing for audio or documents beyond text and images?

Beyond text and image inputs, some LLM providers support audio transcription and document (PDF) understanding as native model inputs. LangChain4j exposes these through additional Content types in the UserMessage builder, following the same pattern as ImageContent.

Audio input — For providers that support audio understanding (like OpenAI GPT-4o Audio or Google Gemini), AudioContent wraps a base64-encoded audio clip with a MIME type:

byte[] audioBytes = Files.readAllBytes(Path.of("customer-call.mp3"));
String base64Audio = Base64.getEncoder().encodeToString(audioBytes);

UserMessage message = UserMessage.from(
    AudioContent.from(base64Audio, "audio/mp3"),
    TextContent.from("Summarize the key complaints in this customer call recording.")
);

PDF / document input — Some providers (Anthropic Claude, Gemini) accept raw PDF bytes as input, allowing the model to read and understand the document structure natively rather than extracting text first:

byte[] pdfBytes = Files.readAllBytes(Path.of("contract.pdf"));
String base64Pdf = Base64.getEncoder().encodeToString(pdfBytes);

UserMessage message = UserMessage.from(
    TextContent.from("Identify all payment terms in this contract."),
    PdfFileContent.from(base64Pdf)  // provider-specific support required
);

Important caveat: multi-modal support beyond text and images is provider-specific. Before using AudioContent or PdfFileContent, verify that your configured model and LangChain4j provider module version support it. Using these content types with a model that does not support them results in an API error from the provider. Always check the LangChain4j integration page for your provider for the current supported content types.