How does LangChain4j support vision (multi-modal) LLMs that accept images as input?

Multi-modal LLMs like GPT-4o, Claude 3, and Gemini can process images alongside text. In LangChain4j, image input is handled through the UserMessage content builder, which accepts a list of Content objects — combining TextContent and ImageContent in a single user turn.

// Pass an image URL
UserMessage message = UserMessage.from(
    TextContent.from("What defects do you see in this product image?"),
    ImageContent.from("https://cdn.example.com/product-photo.jpg")
);

AiMessage response = chatModel.generate(List.of(message)).content();

// Or pass Base64-encoded image data (for local files)
byte[] imageBytes = Files.readAllBytes(Path.of("screenshot.png"));
String base64 = Base64.getEncoder().encodeToString(imageBytes);

UserMessage visionMessage = UserMessage.from(
    TextContent.from("Describe any errors shown in this screenshot"),
    ImageContent.from(base64, "image/png")
);

Vision capabilities also work with AI Services. You can define a method that accepts a UserMessage directly, or use @UserMessage with image parameters:

interface ImageAnalyzer {
    @UserMessage("Analyze this image and describe what you see.")
    String analyze(UserMessage messageWithImage);
}

Important considerations: not all models in the same provider family support vision (e.g., GPT-3.5 cannot process images; GPT-4o can). Check that your configured model name is a vision-capable variant. Image inputs consume significantly more tokens than text, which affects both cost and context window usage — high-resolution images can consume thousands of tokens depending on the model's tile-based processing strategy.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.