How Does AI Know What's Inside an Image? Meet Vision Language Models

Do you interact with AI models where you provide an image and they can understand and respond to it?

Have you ever wondered what these models are and what is happening behind the scenes?

This is where VLMs, or Vision-Language Models, come in.

A VLM combines vision and language capabilities, allowing it to process visual information along with text.

A simple way to think about it is:

LLM → Primarily works with text

VLM → Works with images and text

This is a simplified way of looking at it, since VLM architectures can vary. Under the hood, a VLM often combines a vision encoder, a connector or projector, and a language model.

So, how does a VLM actually "see" an image?

To understand that, we'll take a closer look at the vision encoder.

In this article, we'll explore how this works and build a quick CLIP-based demo that you can try out yourself.

How Does a VLM Actually See?

Let's start with something familiar. As you know, an LLM works with tokens.

But an image isn't naturally made up of language tokens.

So, we need a way to convert the visual information in an image into a representation that the language model can understand and work with.

A simplified view of the process looks like this:

Image
  ↓
Vision Encoder
  ↓
Visual Representations
  ↓
Vision-Language Connector
  ↓
Language Model
  ↓
Text Response

Let's break this down step by step.

Vision Encoder

The first important component is the Vision Encoder.

Its job is to process an image and convert the visual information into numerical representations that the rest of the model can work with.

Depending on the vision encoder architecture, the image may first be divided into smaller regions called patches.

For example:

Each of these patches, such as P1, P2, P3, and so on, is converted into a numerical representation. The vision encoder then processes these representations to capture useful visual patterns and relationships within the image.

These visual representations can capture information about things such as shapes, objects, colors, textures, text, spatial relationships, and other visual features.

The exact process depends on the architecture of the vision encoder. Patch-based processing, like the one shown above, is common in Vision Transformers (ViTs) and architectures based on them.

The result is a set of numerical representations that capture information from the image. These representations can then be passed to the next component, the Vision-Language Connector, which helps make them suitable for the language model.

You can think of this as a kind of translation step: we are converting visual information into a numerical representation that the language model can work with.

Connector and Language Model

Now we have seen the vision encoder and how it processes visual information.

What we generated so far can be thought of as the raw materials. The next stages use these representations to actually generate a response.

The connector takes the visual representations and makes them suitable for the language model, which can then use them to generate the final text response.

Visual Representations
        ↓
    Connector
        ↓
  Language Model
        ↓
   Text Response

So, the connector acts as the bridge between the vision encoder and the language model, allowing the visual information to eventually become a natural language response.

Seeing CLIP in Action

Now let's try something real.

Let me introduce you to CLIP, the model we'll be using to demonstrate how a vision encoder processes an image.

If you ask what CLIP is, it's basically a model that learns the relationship between images and text. CLIP was developed by OpenAI and trained on a large collection of images and their corresponding text descriptions.

It has two main components: an image encoder and a text encoder.

The image encoder processes an image and produces an embedding, while the text encoder processes text and produces an embedding as well. We can then compare these embeddings to determine how closely the image and text are related.

For now, let's focus on the image encoder.

We'll give an image to CLIP and see what happens behind the scenes.

Step 1: Patching

The first step is to divide the image into smaller pieces called patches.

Here, we divide the image into a 7 × 7 grid, giving us a total of 49 patches.

Each square in the grid represents one patch of the original image.

At this stage, we are simply splitting the image into these smaller patches. In the next step, each patch will be converted into a vector, which is a numerical representation that the vision encoder can work with.

So, instead of looking at the entire image as one big piece, the model starts by breaking it down into 49 smaller pieces.

Step 2: Vision Encoding

Now that we have our 49 patches, we need to turn them into something the vision model can understand.

The 49 patches are passed through the vision encoder, which processes them and produces a numerical representation for each patch.

In our example, each patch is represented by a vector containing 768 numbers.

So we go from:

Image
↓
7 × 7 grid
↓
49 patches
↓
49 vectors

The image above visualizes these 49 vectors as squares. The size and color of each square represent the magnitude of that patch's vector.

A larger, redder square means the vector has a larger magnitude, while a smaller, bluer square means it has a smaller magnitude.

It's important to note that these colors do not mean that the model has identified something specific in that patch. For example, a red square does not mean "the model found a dog here."

At this stage, we have 49 numerical representations, one for each patch.

These representations are what we will work with in the next step.

Step 3: Text Encoding

So far, we have seen how CLIP processes the image and turns it into numerical representations.

But CLIP doesn't work with images alone. Remember, CLIP is trained to understand the relationship between images and text.

So, let's now do the same thing with some text.

For this example, we'll use the sentence:

"A dog sitting, holding a flower"

Before the text can be processed by the model, it is first broken down into smaller pieces called tokens.

Below, you can see how our sentence is split into tokens:

These tokens are then passed through CLIP's text encoder, which converts the text into a numerical representation, or text embedding.

So, we now have:

Image ──> Image Encoder ──> Image Embedding

Text  ──> Text Encoder  ──> Text Embedding

Now that we have an embedding for both the image and the text, we can compare them and see how closely they are related.

Step 4: Comparing the Image and Text

Now we have two different embeddings:

• An image embedding produced by the image encoder
• A text embedding produced by the text encoder

The interesting part is that CLIP is designed to place related images and text closer together in the same embedding space.

So, we can compare the image embedding with the text embedding and calculate a similarity score.

For our example, CLIP produces a similarity score of 0.2557 between the image and the text:

"A dog sitting, holding a flower"

A higher similarity score generally means that the image and text representations are more closely aligned.

The important thing to understand is that CLIP isn't generating this sentence from the image. Instead, we're giving CLIP both the image and the text and asking:

"How closely do these two representations match?"

This ability to compare images and text in the same embedding space is what makes CLIP useful for tasks such as zero-shot classification, image search, and image-text retrieval.

Step 5: Relevance Heatmap

Now, let's take a look at the heatmap below.

The warmer areas indicate regions of the image that have a higher relevance to the text we provided.

In this example, the warmer regions are concentrated around the dog, which makes sense because our text describes:

"A dog sitting, holding a flower"

This gives us a visual way to see which parts of the image are contributing more strongly to the image-text relationship.

It's important to remember that this heatmap is a visualization of the model's representations and similarity, not a direct explanation of what the model "thinks" or a precise object detector.

And that's our quick demonstration of how CLIP can connect visual and textual information.

Now, let's explore one more thing: zero-shot classification.

Zero-Shot Classification

Another thing that CLIP is capable of is zero-shot classification.

If you don't know what zero-shot classification means, it is basically the ability to classify something into categories without being specifically trained for those categories as a classification task.

Let's try it with an example.

Here, I have an image of a dog and a set of possible categories:

Dog
Bird
Cat
Car
Beach

CLIP compares the image with each of these text labels and calculates a similarity score for each one.

As you can see, "dog" gets the highest score, meaning the image is most closely aligned with that category.

The interesting part is that we didn't train a new classifier specifically to distinguish between dogs, birds, cats, cars, and beaches. CLIP can use its existing understanding of the relationship between images and text to perform the classification.

VLM vs. Multimodal LLM

You may also come across another term that is closely related to VLMs:

Multimodal LLM

So, what's the difference?

The terminology isn't perfectly standardized, but a useful way to think about it is:

VLM usually refers to models that focus specifically on connecting vision and language.

Examples include:

• Qwen-VL
• Qwen2.5-VL
• Llama Vision
• Pixtral

Multimodal LLM is a broader term for language models that can work with multiple types of input or output, such as:

Text
Images
Audio
Video

So, conceptually, you can think of a VLM as a type of multimodal model that focuses on vision and language.

In practice, though, the terminology can overlap. A model may be described as both a VLM and a multimodal LLM depending on how its architecture and capabilities are being discussed.

Try It Yourself

The project we just demonstrated is available on GitHub for you to try yourself.

Clone the repository, follow the instructions, and give it a try!

GitHub Repository

Wrapping Up

When we talk about Vision-Language Models, the bigger picture looks something like this. We've covered the main stages throughout this article:

Image
  ↓
Vision Encoder
  ↓
Visual Representations
  ↓
Connector / Projector
  ↓
Language Model
  ↓
Response

The vision side processes the image and produces visual representations.

The connector helps make these representations usable by the language model.

The language model then uses this visual information along with the text input to generate a response.

VLMs become especially useful when text alone isn't enough to describe what we want an AI system to understand.

Instead of describing a UI design, diagram, screenshot, chart, or photograph entirely through text, we can simply provide the image.

In that sense, VLMs give AI systems a way to work with visual information alongside language, making it possible to interact with information that would otherwise have to be described using words.