Multimodality

Humans perceive the world through multiple channels. We see, hear, and read, and we automatically connect these impressions. Early AI programs could not do this. They were fixed on a single type of input: either text, images, or sound. Multimodality describes a program’s ability to process several of these channels together. One channel is called a “modality,” hence the name.

What a second sensory channel makes possible

Many tasks simply cannot be solved with text alone. Anyone who wants to know why a machine won’t start prefers to send a photo. Anyone who wants a math problem from a workbook explained photographs the page. A system that only reads text is useless here. This is exactly why multimodality has become so important for providers of AI products.

There is also a practical point: a great deal of the world’s knowledge is not written in sentences. It is contained in diagrams, blueprints, X-ray images, tables captured as photos, or in video recordings. A model that only understands written language is cut off from these amounts of data. Multimodal systems can make use of them as well.

It is important to distinguish this from a mere stringing-together of tools. One could also have an image-recognition program describe a photo and then pass that description on to a language program. This works, but loses details. True multimodality means that a model factors in the image itself, rather than merely reading a summary of it.

Translating images and sounds into a common language

An AI model internally computes only with lists of numbers. Text is converted into such number lists before processing, similar to how music is turned into musical notation. The trick with multimodality is translating images and sounds into that same notation. Then all inputs exist in the same format, and the model does not need to distinguish where they came from.

For images, a separate component called the encoder takes care of this. It breaks the image down into small patches and describes each patch with a list of numbers. These lists are passed to the actual language model together with the text. For sound, a similar approach is used: the recording is split into short segments and likewise converted into numbers.

For the result to be coherent, training requires matching example pairs. Millions of images with corresponding descriptions are often already enough. From these pairs, the model learns which image features belong to which words. A common misconception is that a multimodal model truly “sees.” It recognizes patterns in numbers that arose from pixels, and in doing so it also makes typical mistakes, for instance when counting many objects.

Multimodal features in today’s apps

Multimodality is most visible in the large chat assistants. You can upload a photo and ask questions about it, or speak directly into a conversation. Phone cameras use this too: the camera app recognizes text in the image, translates signs, or explains plants. In cars, systems jointly evaluate camera images and radar data.

In business news, the term usually comes up with new model versions. Manufacturers then emphasize that their model is “natively multimodal,” meaning it was trained with image and sound from the very start. This is considered a selling point compared to older models, where image understanding was added on afterward.

The next step is called multimodality on the output side as well. Such systems respond not only with text but generate images, speech, or short videos. For professions such as graphic design, translation, or customer service, this is the reason why this technology is being discussed so much.