# What is a VLM: how vision-language models read images and video

> A vision encoder, a small projector and a language model: how VLMs are built, what CLIP, Flamingo and LLaVA showed, and where they still miscount and hallucinate.

URL: https://claudegpt.si/news/what-is-a-vlm-vision-language-model  
Published: 2026-11-16T10:58:02.892Z  
Author: ClaudeGPT Si Newsroom  
Section: Research  
Topics: Multimodal models, Research, Language models, Research papers, AI hallucination, OCR  
Publisher: ClaudeGPT Si

## Key takeaways

- A VLM joins a vision encoder, a projector and a language model so it can answer text questions about images and video.
- LLaVA showed in 2023 that one trainable linear layer between CLIP and the Vicuna language model was enough to build an image chat assistant.
- In one 2024 test, four leading VLMs averaged 58.07% on simple tasks such as counting overlapping circles, and object hallucination remains common.

The short answer to "what is a VLM": a vision-language model is an AI model that takes images or video along with text and answers in text. Most are built from three parts: a vision encoder that turns the picture into numbers, a small projector that translates those numbers into what a language model reads, and a [large language model](https://claudegpt.si/news/what-is-an-llm) that writes the answer.

The idea spread because it is cheap. Instead of training one giant model from scratch, researchers attached a pretrained image model to a pretrained text model and taught the two to talk. Chat assistants that accept photos are VLMs too: a 2024 study tested GPT-4o, Gemini 1.5 Pro and Claude 3.5 Sonnet as VLMs, though their makers don't publish how those models are wired inside.

## How a VLM works: encoder, projector, language model

**The vision encoder** cuts the picture into small patches and turns each one into a vector, a list of numbers describing what is in it.

The encoder most open VLMs started from is CLIP, from OpenAI's [February 2021 paper](https://arxiv.org/abs/2103.00020). CLIP was trained on 400 million image and caption pairs from the internet with one task: pick which caption goes with which image. Without seeing any of ImageNet's 1.28 million labeled training images, it matched the accuracy of the original ResNet-50, a classifier trained on all of them. OpenAI released the weights, so anyone could reuse the encoder.

**The projector** is the bridge. A language model reads [tokens](https://claudegpt.si/news/what-is-a-token-in-ai), the word pieces it works in, and the encoder's vectors mean nothing to it. The projector maps each image vector into the language model's word-embedding space. After that, the image looks to the language model like a few hundred extra words at the start of the prompt.

**The language model** reads those image tokens with your question and generates the answer one token at a time, as it would for a text-only prompt.

### LLaVA showed how little the bridge needs

LLaVA, from researchers at the University of Wisconsin-Madison, Microsoft Research and Columbia University, is the clearest example. The April 2023 paper [Visual Instruction Tuning](https://arxiv.org/abs/2304.08485) connected CLIP's ViT-L/14 encoder to the Vicuna language model with one trainable linear layer, trained in two stages:

1. **Alignment.** Encoder and language model stayed frozen. Only the projector learned, on 595,000 image and caption pairs. The authors call this training "a compatible visual tokenizer for the frozen LLM."
2. **Instruction tuning.** Text-only GPT-4 wrote 158,000 conversations, descriptions and reasoning questions about images, and the projector and language model were trained on them.

The result could chat about pictures, and fine-tuned on ScienceQA with GPT-4's help it reached 92.53% accuracy, the best score then. [LLaVA-1.5](https://arxiv.org/abs/2310.03744) swapped the linear layer for a small multilayer network and trained on 1.2 million public examples in about a day on eight A100 GPUs.

### Flamingo took a different route

DeepMind's [Flamingo](https://arxiv.org/abs/2204.14198), from April 2022, also kept its vision encoder and language model frozen. A module called the Perceiver Resampler squeezes any image or video into a small fixed number of visual tokens, and new cross-attention layers between the language model's layers let the text look at them. Trained on web pages that mix text and images, Flamingo learned from a few examples in the prompt. On 6 of 16 image and video tasks it beat models fine-tuned on about 1,000 times more task data, using only 32 examples.

## What VLMs are used for

| Task | Example |
| --- | --- |
| Visual question answering | "How many people are in this photo?" |
| Captioning | Describing a scene or a video clip |
| Reading text in images | Receipts, signs, forms, screenshots |
| Document and chart questions | A figure from a scanned report or a chart |
| Science and math with figures | ScienceQA, MathVista |
| Video understanding | What happens in a clip, in what order |

The same design also sits under vision-language-action models for robots, which add actions to the output (see [PDRVLA](https://claudegpt.si/news/pdrvla-vision-language-action-model)).

## Where VLMs go wrong: hallucination, counting, OCR

VLMs answer fluently whether or not they saw correctly. Three weak spots have been measured.

**Hallucination.** A [2023 study](https://arxiv.org/abs/2305.10355) found the VLMs it tested "mostly suffer from severe object hallucination": they describe objects that aren't in the image. The invented objects tend to be ones common in training instructions, or ones that usually appear next to objects that are in the picture.

**Counting and fine geometry.** In [Vision language models are blind](https://arxiv.org/abs/2407.06581), four leading VLMs got seven tasks easy for people: do two circles overlap, how many times do two lines cross, how many rings are in an Olympic-like logo. They averaged 58.07%. Claude 3.5 Sonnet did best at 77.84%. The encoder held enough information to answer; the language model failed to turn it into the right words.

**Reading text.** [OCRBench](https://arxiv.org/abs/2305.07895), built from 29 datasets, found VLMs weak on handwriting, non-English text, strings that aren't real words (serial numbers, codes) and math expressions. On invoices and IDs, that is often the text that matters.

The failures share a cause: the language model answers, and when the visual evidence is weak, it falls back on what text usually says. Where a wrong count or digit costs money, check the output.

## This week's research: faster VLMs, and video that moves

Two papers posted on arXiv on November 13, 2026 work on opposite ends of the pipeline. Both report the authors' own numbers, not yet independently reproduced.

**MMFlash makes VLMs answer faster.** [MMFlash](https://arxiv.org/abs/2611.18482), from the National University of Singapore, applies speculative decoding to VLMs: a small drafter guesses several tokens and the big model checks them in one pass, so the output is the same, only faster. The authors found drafters miss the words that depend on the image. For the least image-dependent quarter of tokens, 46% of drafts were accepted; for the most image-dependent quarter, 22%. MMFlash weights those tokens more in training and uses sparse attention for long video. On Qwen3.5 models it reports average speedups of 3.00 to 3.05 times on images, 2.33 to 2.71 times on long video, and up to 3.58 times, beating EAGLE-3 and DFlash. It had 28 upvotes on Hugging Face on November 16, and the [code is on GitHub](https://github.com/fscdc/MMFlash).

**V-JEPA-LM makes video VLMs read motion.** [V-JEPA-LM](https://arxiv.org/abs/2611.17831), from Hanyang University in Seoul, starts from a gap. A simple probe reads an object's speed from the frozen features of the V-JEPA video encoder 87% of the time, yet a VLM built on it answered at chance. Captions name a few things in each clip, so the model learned to look those up rather than watch what changes. The fix was training clips played forward, reversed, mirrored or re-timed, where only the motion decides the answer. Trained on 4.53 million public samples, the 4B model averages 72.1 across eleven video benchmarks, against 61.1 for the best open 4B model and 64.8 for the best open 8B. [Weights and inference code](https://github.com/kkt94/Llama-V-JEPA-LM) are out.

The 2023 recipe still holds. The work now is making it cheaper to run and getting the language model to use what the encoder sees.

## Frequently asked questions

### What is the difference between a VLM and an LLM?

An LLM reads and writes only text. A VLM adds a vision encoder and a projector in front of a language model, so images and video become extra tokens the language model can read; the answer still comes out as text.

### What does VLM mean?

VLM stands for vision-language model, sometimes written visual language model, as in DeepMind's Flamingo paper. It means a model that takes images or video plus text as input and produces text.

### Are there open-source VLMs?

Yes. LLaVA released its code, training data and model, OpenAI released CLIP's weights, and the authors of V-JEPA-LM, posted on November 13, 2026, have released its weights and inference code.

### Can a VLM do OCR?

VLMs read ordinary printed text well, but the OCRBench study found them weak on handwriting, non-English text, strings that are not real words and math expressions, so important fields should be checked.

## Sources

- [Learning Transferable Visual Models From Natural Language Supervision (CLIP)](https://arxiv.org/abs/2103.00020), arXiv (OpenAI)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://arxiv.org/abs/2204.14198), arXiv (DeepMind)
- [Visual Instruction Tuning (LLaVA)](https://arxiv.org/abs/2304.08485), arXiv
- [Improved Baselines with Visual Instruction Tuning (LLaVA-1.5)](https://arxiv.org/abs/2310.03744), arXiv
- [Evaluating Object Hallucination in Large Vision-Language Models (POPE)](https://arxiv.org/abs/2305.10355), arXiv
- [Vision language models are blind](https://arxiv.org/abs/2407.06581), arXiv
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://arxiv.org/abs/2305.07895), arXiv
- [MMFlash: Faster Speculative Decoding for Vision-Language Models](https://arxiv.org/abs/2611.18482), arXiv (National University of Singapore)
- [MMFlash code](https://github.com/fscdc/MMFlash), GitHub
- [MMFlash on Hugging Face Papers](https://huggingface.co/papers/2611.18482), Hugging Face
- [V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations](https://arxiv.org/abs/2611.17831), arXiv (Hanyang University)
- [Llama-V-JEPA-LM code and models](https://github.com/kkt94/Llama-V-JEPA-LM), GitHub
- [Tackling multiple tasks with a single visual language model](https://deepmind.google/discover/blog/tackling-multiple-tasks-with-a-single-visual-language-model/), Google DeepMind
