Natively multimodal model

Computer programs that understand language were long fed only with written text. If you wanted to show them images or sounds as well, you attached extra programs in front. An image was thus first translated into a description by its own dedicated program, and this description was then read by the language program. A natively multimodal model does things differently: it learns text, images, sound, and video together from the start, within a single system. “Native” here means “from birth” — the capability was not bolted on afterward. “Modality” is simply the technical term for a type of information, that is, speech, writing, image, or sound.

What gets lost in the detour through text

The older construction is like a conversation with an interpreter who only passes along keywords. Whatever is missing from those keywords is gone forever afterward. When an add-on program converts a voice recording into text, tone of voice, volume, and pauses disappear. The language model can then no longer tell whether someone sounded angry, uncertain, or ironic.

The same holds for images. An automatic image caption might say “a chart with a curve,” but not the exact numbers on the axis. A natively multimodal model sees the pixels itself and can therefore read values off a graph or decipher handwritten notes. That is exactly what users need when they upload a photo of their homework sheet.

Then there is speed. Every intermediate stage costs time, usually a few hundred milliseconds. For a voice assistant that is supposed to feel like a real conversation, that is too much. A single system that hears directly and answers directly can respond with almost no delay and can even be interrupted mid-sentence.

Everything becomes chains of numbers

The trick lies in a shared internal language within the model. Every input is broken down into small chunks called tokens, and each chunk is represented as a long list of numbers. For text, such a chunk is a syllable or a short word. For an image, it is a small image patch; for sound, a fraction of a second. In the end, all chunks look the same to the model: they are simply numbers.

That is why the same computing machine inside can handle everything. It is trained to predict the next chunk in a sequence of chunks. Whether a word follows an image or a sound follows a word makes no difference to it. In this way, the model learns connections between modalities on its own: that the word “dog” is associated with a certain appearance and a certain sound.

Training for this requires suitable data — videos with an audio track, images with captions, films with subtitles. Such pairs are the real bottleneck. Incidentally, a common misconception is that any model with image capability is natively multimodal. Many systems are not — they merely combine several building blocks cleverly, and this is often revealed by the fact that they fail at certain combinations.

Camera chats and real-time translation

The difference is most noticeable in voice modes within chat apps. You point your phone’s camera at a shelf, speak a question, and immediately get a spoken answer back. Live translation that preserves emphasis and speech pauses also only works this way. Providers like OpenAI, Google, and Meta explicitly market their newer models with this approach.

In news items and product announcements, words like “omni,” “natively multimodal,” or “vision model” show up for this. That is marketing, but it usually does describe exactly this design. For companies, it is appealing because a maintenance team can send a photo of a broken machine and explain at the same time what sound it makes.

There are limits nonetheless. Video devours a great many chunks and thus a lot of computing power, which is why long films are usually only sampled rather than analyzed in full. And a model that is somewhat capable at everything is not automatically better than a specialized program built for a single task.