Spring / Spring AI interview questions
How does Spring AI support multimodal inputs such as images?
Multimodal support in Spring AI means sending both text and non-text content — images, audio — to models that can process them (GPT-4o, Claude 3, Gemini, Llama 3.2 Vision). The UserMessage class accepts a list of Media objects alongside the text content, and each Media wraps a MIME type plus either raw bytes or a URL reference to the image.
// Load image from classpath Resource imageResource = new ClassPathResource("screenshot.png"); UserMessage message = new UserMessage( "Describe what is wrong in this UI screenshot.", List.of(new Media(MimeTypeUtils.IMAGE_PNG, imageResource)) ); ChatResponse response = chatModel.call(new Prompt(message)); String description = response.getResult().getOutput().getContent();
With ChatClient the fluent API makes it equally clean:
String analysis = chatClient.prompt() .user(u -> u.text("What Java exception is shown in this stack trace image?") .media(MimeTypeUtils.IMAGE_PNG, new ClassPathResource("stacktrace.png"))) .call().content();
Spring AI passes the image to the provider using whichever encoding that provider requires — OpenAI uses base64 JSON or URL references inside the messages array; Google uses the Vertex multimodal parts API — but the application code is the same regardless. Not all providers support all media types. OpenAI and Google Vertex support PNG/JPEG images; some providers also support PDF documents or audio clips. Always check the provider's Spring AI documentation for supported MIME types before assuming portability.
More Related questions...
