Local multimodal models: the VRAM arithmetic on a 16 GB card

You have a 16 GB card and you want to show a screenshot to a model running on your own machine. The question is not whether the model exists, several do: it is whether weights, vision projector, KV cache and image tokens all fit in those 16 GB with a context window worth having. Here is the arithmetic, using the exact byte counts published by Qwen, Ollama and the llama.cpp repository, so you can redo it for your own card.

Key takeaways

• A multimodal model is three pieces: vision encoder, projector and language model. Only the third one is re-read for every token generated.
• Qwen3-VL 8B weights at Q4_K_M come to 4.68 GiB, and the vision projector is a separate 1.08 GiB file that almost nobody counts.
• That model’s KV cache costs 144 KiB per token. At 8,192 tokens that is 1.13 GiB; at the native 262,144 it would be 36 GiB.
• A 1920 × 1080 screenshot becomes 2,025 visual tokens: 285 MiB of cache and 2,025 tokens of context spent before you type the question.
• 16 GB comfortably holds 7B to 12B models with 8K to 32K of context. What it does not hold is long context, video, or a 24B with room to work.

How a multimodal model is put together

A vision language model is not a text model with a camera bolted on. It is three blocks in a chain. The first is a vision encoder, almost always a Vision Transformer, which cuts the image into patches and produces one vector per patch. The second is a projector, a small layer that translates those vectors into the language model’s embedding space. The third is the language model itself, which receives those vectors interleaved with the text and writes the answer.

That separation has a very concrete practical consequence: all three blocks take memory, but they do not behave the same way. The encoder and the projector run once per image, during prefill. The language model is re-read in full for every token it writes. That is why the size of the weights file governs speed, while the encoder governs how many images you can feed before you run out of room.

In Qwen3-VL the structure is explicit right down to the files you download. The official GGUF repository publishes the model and the projector separately, and llama.cpp loads them separately too, with the --mmproj argument. In MiniCPM-V the split is even more visible: the Ollama manifest lists the projector as a layer of its own, 0.97 GiB, distinct from the model’s 4.13 GiB.

The arithmetic for Qwen3-VL 8B

Four things share the card: the language model weights, the vision encoder with its projector, the KV cache, and the CUDA context plus activation buffers, which come to about half a gigabyte. The first two are files you can weigh, and how they change with the quantisation you pick is covered in the article on model quantisation with llama.cpp. The third you have to compute, and it comes out of four numbers in the model’s config.json: 36 layers, 8 key and value heads, head dimension 128, and two bytes per value if you keep it in half precision.

# Qwen3-VL-8B-Instruct, values taken from its config.json
layers, kv_heads, head_dim, bytes_per_value = 36, 8, 128, 2

per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
print(per_token)                      # 147456 bytes = 144 KiB per token

for ctx in (8192, 32768, 262144):
    print(ctx, round(per_token * ctx / 2**30, 2), "GiB")
    # 8192    1.13 GiB
    # 32768   4.5  GiB
    # 262144  36.0 GiB

With that the table fills itself in. The first two sizes are what the Hugging Face API returns for the files in Qwen’s official repository, not estimates.

Adding weights, projector and an 8K cache gives 6.89 GiB, which with the CUDA context lands around 7.5 GiB. Half the card to spare. At 32K of context the total rises to 10.26 GiB and still fits. At 64K the cache alone asks for 9 GiB and the total brushes 15 GiB: that is where a 16 GB card says no. And the 256K native context the model advertises is simply out of reach on consumer hardware, at 36 GiB of cache alone.

One check that gives confidence in these numbers: 4.68 plus 1.08 is 5.76 GiB, and the blob Ollama downloads for the qwen3-vl:8b tag measures 5.72 GiB. Two independent sources agreeing to within one percent.

What a screenshot actually costs

This is where the arithmetic gets interesting, because an image’s cost is not fixed: it depends on the pixel count. Qwen3-VL’s preprocessor_config.json declares a patch size of 16 and a spatial merge of 2, so each visual token summarises a 32 × 32 pixel block.

# preprocessor_config.json: patch_size 16, merge_size 2
px_per_token = (16 * 2) ** 2          # 1024 pixels per visual token

print(1920 * 1080 // px_per_token)    # 2025 tokens
print(3840 * 2160 // px_per_token)    # 8100 tokens
print(16777216 // px_per_token)       # 16384, the processor's ceiling

That last number is no accident. The processor’s size field declares a maximum of 16,777,216 pixels, which divided by 1,024 gives exactly 16,384 tokens, and a minimum of 65,536 pixels, which gives exactly 64. The values are exact multiples, which confirms the 32 × 32 block is the right one.

Translated into memory: a 1080p screenshot takes 2,025 visual tokens, which at 144 KiB per token is 285 MiB of KV cache. A 4K capture goes to 8,100 tokens and 1.11 GiB. It is not that it will not fit; it is that if you hand it four 4K screenshots you have eaten 32,400 tokens of context and 4.4 GiB of cache before the model writes a single word.

Gemma 3 solves the same problem from the opposite direction. Its encoder works at a fixed 896 × 896 resolution and compresses every image into a fixed number of vectors. The Google DeepMind technical report puts it plainly: "We reduce the inference cost of image processing by condensing the vision embeddings into a fixed size of 256 vectors". Two hundred and fifty-six tokens per crop, always, whatever the input resolution. It is predictable and cheap, and in exchange it loses fine detail on wide images unless the adaptive cropping they call Pan and Scan kicks in.

Which models really fit in 16 GB

Every size in this table comes from querying the Ollama registry manifests directly, not from the web page or somebody’s roundup.

Two more fit without trouble outside the table: llama3.2-vision:11b at 7.28 GiB and minicpm-v:8b at 4.13 GiB of model plus 0.97 GiB of projector. The last row marks the real limit: a quantised 24B fits on the card, but with under 2 GiB free the context you have left is barely worth having.

How to run them

With Ollama the shortest route is two commands, and the image path goes inside the message itself. If you are starting from nothing, setup is covered in the Ollama installation guide.

ollama pull qwen3-vl:8b
ollama run qwen3-vl:8b "Describe the error shown in ./screenshot.png"

  # inside the interactive session, to pin the context:
  # /set parameter num_ctx 8192

That num_ctx is not a minor detail. Ollama starts with a default context far below the 262,144 tokens the model accepts, and a 1080p screenshot already takes 2,025 of them. If the model seems to forget the image halfway through a conversation, this is almost always why.

With llama.cpp the control is finer because you load both files by hand. The libmtmd library is what provides multimodal support, and llama-cli, llama-server and llama-mtmd-cli all accept the --mmproj argument.

llama-mtmd-cli 
  -m Qwen3VL-8B-Instruct-Q4_K_M.gguf 
  --mmproj mmproj-Qwen3VL-8B-Instruct-F16.gguf 
  --image screenshot.png 
  -p "What error does this screenshot show?" 
  -c 8192 -ngl 99

By default the projector is offloaded to the GPU as well. If you are tight on VRAM, --no-mmproj-offload keeps it on the CPU and hands that gigabyte back, at the cost of a slower prefill. To poke around without a command line, LM Studio loads the same GGUFs with their projector.

How fast they actually go

Time to be honest: we do not have a 16 GB NVIDIA card to measure on, so we are not going to invent a figure. What does exist is a measurement published in the CUDA performance thread of the llama.cpp repository itself. An RTX 5060 Ti 16 GB, running llama-bench with full GPU offload, gives 93.46 tokens per second of generation and 4,195 tokens per second of prefill on Llama 2 7B at Q4_0 (3.56 GiB of weights), with flash attention on.

From there you can estimate, without measuring, what to expect from Qwen3-VL 8B. Generation is bound by memory bandwidth, because every token forces a re-read of the language model weights. Scaling by the size ratio, 93.46 times 3.56 over 4.68 gives roughly 71 tokens per second. That is an estimate, not a measurement: Q4_0 and Q4_K_M use different kernels, and the projector adds a Vision Transformer pass that llama-bench does not time.

For prefill the sum is more direct. At 4,195 tokens per second, the 2,025 tokens of a 1080p screenshot take half a second to go in, plus whatever the vision encoder costs. So the first answer takes a second or so to start and then flows. Enough to work with. A long way from a hosted model, but that was never in doubt.

Where they get it right and where they fail

Qwen says of its own model that "It can now generate code from images or videos, for example, turning a design mockup into Draw.io, HTML, CSS, or JavaScript code". That is the vendor’s claim, not an independent result, and it is worth treating as such. What is consistent with anyone’s experience of these models is that they do well at what you might call medium-grain understanding: reading a console error message, describing the structure of an interface, extracting the shape of an architecture diagram, transcribing large clean text.

Where they break is fine detail, and the reason shows up in the arithmetic itself. If each token summarises a 32 × 32 pixel block, a 12-pixel interface font puts four or five characters inside a single visual token. The model is not reading those letters, it is guessing which word fits that smudge. That is why dense tables come back with cells shuffled and small figures come back wrong.

Then there is the structural problem, which more resolution does not fix. The work by Rahmanzadehgervi and colleagues, published in July 2024, put four leading models through seven elementary visual tasks, of the "do these two circles touch" and "count the rings in an Olympic-style logo" variety. The average was 58.07 percent and the best of them managed 77.84, against the 100 you would expect from a person. Their linear probing showed the vision encoder did hold the information; it was the language model that could not read it back. If that happens to big hosted models, do not expect an 8B on your desk to give you reliable coordinates.

The practical conclusion is simple: ask them for description and structure, not for measurements or exact positions. The same thing already came up when discussing GPT-4o’s native multimodality and Gemini 2.5’s context scaling, but with less margin here.

When 16 GB is not enough

There are three situations where the card runs out and no amount of parameter tuning fixes it. The first is long context: above 64K tokens the KV cache eats the card on its own. The second is video, because a handful of frames multiplies the image cost by the frame count. The third is 24B-class models with room to work.

The clearest example of that third case was published on 9 August 2026: Meta’s Muse-Glimmer-30B, Apache 2.0, 29.6 billion parameters of which 1.8 billion are a ViT-G/14 encoder. Its most compressed GGUF build weighs 15.61 GiB and its projector another 1.30 GiB, so it totals 16.91 GiB and misses a 16 GB card by a little over a gigabyte. Its card places it in a 24 or 32 GB envelope, which is where it belongs.

That model also teaches something the download table does not say: it has 52 layers but only two key and value heads, so its KV cache costs 52 KiB per token, barely a third of what Qwen3-VL 8B costs, and that model is four times smaller. File size is not the only axis in this sum, and sometimes it is not even the deciding one.

There are three ways out too, in order of cost. The cheap one is dropping a size: qwen3-vl:4b takes 3.07 GiB and leaves 12 GB free for context, and for reading console errors it does the job. The middle one is quantising the KV cache to 8 bits, which halves it, or offloading the projector to the CPU. The expensive one is changing cards, and there it is worth looking first at what alternatives to NVIDIA exist, because above 16 GB the criterion stops being raw power and becomes how much memory you get per euro.

Frequently asked questions

Can I use the 256K context Qwen3-VL advertises?

Not on a consumer card. That model’s KV cache costs 144 KiB per token, so 262,144 tokens would be 36 GiB of cache alone, before the weights. That context is meant for servers with several cards or a lot of unified memory.

Is Q8 worth it over Q4 with 16 GB?

It depends on the context you need. The Q8_0 GGUF of Qwen3-VL 8B weighs 8.11 GiB against 4.68 for Q4_K_M. With the projector and an 8K cache you land at 10.3 GiB, which fits, but you lose half your headroom for context and for images.

Why does my multimodal model take more space than the spec sheet said?

Because the spec sheet usually counts only the language model weights. It leaves out the vision projector, which in Qwen3-VL 8B is a separate 1.08 GiB file, and it leaves out the KV cache, which depends on the context you configured rather than on the model.

Conclusion

The short answer is yes, an 8B multimodal model fits in 16 GB of VRAM with enough context to work, and runs at around 70 tokens per second on a recent card in that class. The long answer is that the number that matters is not the 5 GB of the file, but the 144 KiB per token of the cache and the 2,025 tokens a screenshot takes. With those two figures and a calculator you can decide in a minute which model and which context fit, without trusting anybody’s requirements table. The Spanish version of this article is at Modelos multimodales en local: la cuenta de VRAM en una tarjeta de 16 GB.

Sources

1. Qwen, Qwen3-VL-8B-Instruct model card with its config.json and preprocessor_config.json
2. Qwen, Qwen3-VL-8B-Instruct GGUF repository with the mmproj files
3. Gemma Team, Google DeepMind, Gemma 3 technical report
4. Ollama, qwen3-vl library page
5. llama.cpp, multimodal documentation
6. llama.cpp, performance thread on NVIDIA CUDA
7. Rahmanzadehgervi, Bolton, Taesiri and Nguyen, Vision language models are blind
8. Meta, Muse-Glimmer-30B model card
9. Meta, Muse-Glimmer-30B GGUF repository