AI / LangChain4j interview questions
How does LangChain4j support vision (multi-modal) LLMs that accept images as input?
Multi-modal LLMs like GPT-4o, Claude 3, and Gemini can process images alongside text. In LangChain4j, image input is handled through the UserMessage content builder, which accepts a list of Content objects — combining TextContent and ImageContent in a single user turn.
// Pass an image URL UserMessage message = UserMessage.from( TextContent.from("What defects do you see in this product image?"), ImageContent.from("https://cdn.example.com/product-photo.jpg") ); AiMessage response = chatModel.generate(List.of(message)).content(); // Or pass Base64-encoded image data (for local files) byte[] imageBytes = Files.readAllBytes(Path.of("screenshot.png")); String base64 = Base64.getEncoder().encodeToString(imageBytes); UserMessage visionMessage = UserMessage.from( TextContent.from("Describe any errors shown in this screenshot"), ImageContent.from(base64, "image/png") );
Vision capabilities also work with AI Services. You can define a method that accepts a UserMessage directly, or use @UserMessage with image parameters:
interface ImageAnalyzer { @UserMessage("Analyze this image and describe what you see.") String analyze(UserMessage messageWithImage); }
Important considerations: not all models in the same provider family support vision (e.g., GPT-3.5 cannot process images; GPT-4o can). Check that your configured model name is a vision-capable variant. Image inputs consume significantly more tokens than text, which affects both cost and context window usage — high-resolution images can consume thousands of tokens depending on the model's tile-based processing strategy.
More Related questions...
