How does Spring AI support multimodal inputs such as images?

Multimodal support in Spring AI means sending both text and non-text content — images, audio — to models that can process them (GPT-4o, Claude 3, Gemini, Llama 3.2 Vision). The UserMessage class accepts a list of Media objects alongside the text content, and each Media wraps a MIME type plus either raw bytes or a URL reference to the image.

// Load image from classpath
Resource imageResource = new ClassPathResource("screenshot.png");

UserMessage message = new UserMessage(
    "Describe what is wrong in this UI screenshot.",
    List.of(new Media(MimeTypeUtils.IMAGE_PNG, imageResource))
);

ChatResponse response = chatModel.call(new Prompt(message));
String description = response.getResult().getOutput().getContent();

With ChatClient the fluent API makes it equally clean:

String analysis = chatClient.prompt()
    .user(u -> u.text("What Java exception is shown in this stack trace image?")
               .media(MimeTypeUtils.IMAGE_PNG,
                      new ClassPathResource("stacktrace.png")))
    .call().content();

Spring AI passes the image to the provider using whichever encoding that provider requires — OpenAI uses base64 JSON or URL references inside the messages array; Google uses the Vertex multimodal parts API — but the application code is the same regardless. Not all providers support all media types. OpenAI and Google Vertex support PNG/JPEG images; some providers also support PDF documents or audio clips. Always check the provider's Spring AI documentation for supported MIME types before assuming portability.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.