How does LangChain4j support multi-modal input processing for audio or documents beyond text and images?

Beyond text and image inputs, some LLM providers support audio transcription and document (PDF) understanding as native model inputs. LangChain4j exposes these through additional Content types in the UserMessage builder, following the same pattern as ImageContent.

Audio input — For providers that support audio understanding (like OpenAI GPT-4o Audio or Google Gemini), AudioContent wraps a base64-encoded audio clip with a MIME type:

byte[] audioBytes = Files.readAllBytes(Path.of("customer-call.mp3"));
String base64Audio = Base64.getEncoder().encodeToString(audioBytes);

UserMessage message = UserMessage.from(
    AudioContent.from(base64Audio, "audio/mp3"),
    TextContent.from("Summarize the key complaints in this customer call recording.")
);

PDF / document input — Some providers (Anthropic Claude, Gemini) accept raw PDF bytes as input, allowing the model to read and understand the document structure natively rather than extracting text first:

byte[] pdfBytes = Files.readAllBytes(Path.of("contract.pdf"));
String base64Pdf = Base64.getEncoder().encodeToString(pdfBytes);

UserMessage message = UserMessage.from(
    TextContent.from("Identify all payment terms in this contract."),
    PdfFileContent.from(base64Pdf)  // provider-specific support required
);

Important caveat: multi-modal support beyond text and images is provider-specific. Before using AudioContent or PdfFileContent, verify that your configured model and LangChain4j provider module version support it. Using these content types with a model that does not support them results in an API error from the provider. Always check the LangChain4j integration page for your provider for the current supported content types.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.