How Multimodal Large Language Models See, Think, and Reason

2026-02-13 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Intermediate, medium

Summary

Multimodal Large Language Models (MLLMs) extend transformer capabilities to process visual data alongside text, enabling AI to "see" and reason. This is achieved by treating image patches as "visual tokens" via Vision Transformers (ViT), which are then projected into the LLM's embedding space using linear layers or Q-Formers for efficiency. MLLMs undergo a three-stage training process: Vision-Language Pretraining on image-text pairs, Instruction Tuning with conversational datasets, and Alignment using methods like DPO to ensure helpful and accurate responses. These models demonstrate capabilities such as visual chain-of-thought reasoning, multi-image analysis, document understanding, and video comprehension, with ongoing research addressing challenges like token budget limitations, precise grounding, hallucination, and fine-grained recognition.

Key takeaway

For AI Engineers and Machine Learning Engineers developing new applications, MLLMs offer powerful capabilities for integrating visual and textual reasoning. You should prioritize using existing open-source models like LLaVA and fine-tuning with LoRA for custom use cases, as this significantly reduces development time and computational requirements. Be mindful of the token budget when handling multiple images and employ structured prompt engineering to maximize model performance and minimize hallucinations.

Key insights

MLLMs integrate visual perception into transformers by converting image patches into tokens for sequential processing and reasoning.

Principles

• Treat image patches as visual tokens for transformer input.
• Compress visual tokens for efficiency using Q-Formers.
• Train MLLMs in stages: pretraining, instruction tuning, alignment.

Method

MLLMs process images by chopping them into 16x16 patches, converting these into embeddings via a vision encoder, and then projecting them into the LLM's embedding space for joint text-image reasoning.

In practice

• Use LLaVA, Qwen-VL, or InternVL as starting points.
• Fine-tune MLLMs with LoRA for specific tasks.
• Structure prompts for MLLMs to improve response quality.

Topics

• Multimodal LLMs
• Vision Transformers
• MLLM Training
• Visual Reasoning
• Multimodal Agents

Best for: AI Engineer, Machine Learning Engineer, AI Student

Related on AIssential

• Grandes Modelos de Linguagem Multimodais (MLLMs): Da Teoria \`a Pr\'atica — MLLMs merge LLM language capabilities with multimodal perception, advancing AI's understanding of diverse data.
• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs — Visual latents in MLLMs can be "silenced" by training objectives, but their reasoning capacity can be unleashed at inference time.
• GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents — Multimodal perception must be a core component of agent reasoning, not merely an auxiliary interface.
• Comparative Evaluation of Machine Translation Systems on Images with Text — MLLMs excel in image-to-text translation, outperforming other paradigms through enhanced contextual understanding.
• Social Norm Reasoning in Multimodal Language Models: An Evaluation — MLLMs demonstrate stronger social norm reasoning in text than images, with GPT-4o leading performance.
• Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages — Multimodal LLMs must integrate text, speech, and vision for low-resource languages, moving beyond English-centric benchmarks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.