How much VRAM do you need to run a local LLM?

The internet's answer to this is a table of model sizes. That table is wrong, or at least incomplete, and it is why people download a model that "should fit" and watch it crawl.

Here is the procedure.

Step 1: find out what you actually have

Not what the box said. What the machine reports.

• Nvidia: nvidia-smi — read Total, not Free.
• Apple Silicon: unified memory, shared with everything else the Mac is doing.
• Ollama users: the server log reports the total VRAM it discovered at startup. Use that number, because it is the one your runner will act on.

If the reported figure is far below your card's size, stop here — you have a detection problem, not a sizing problem. See Ollama not using your GPU.

Step 2: subtract the things that are not the model

This is the step everyone skips. Your budget is not "VRAM minus file size".

A useful working rule: leave around 20% of your VRAM free, more if you intend to use a long context. A model that exactly fills your card does not run; it spills.

Step 3: pick the biggest model that fits the remainder

Now use the file size. GGUF filenames encode the quantization, which is what determines size:

These are starting points, not guarantees — architectures vary, and a mixture-of-experts model needs memory for all its experts even though only a few run per token, which makes it a worse fit than its speed suggests.

The rule that beats the table: for a fixed memory budget, a bigger model quantized harder usually beats a smaller model at higher precision. Prefer 13B at Q4 over 7B at Q8.

Step 4: check you were right

Do not trust the arithmetic. Load the model and ask:

ollama ps

NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL
gemma4:latest    c6eb396dbd59    9.6 GB    100% GPU     131072     2 minutes from now

• PROCESSOR says 100% GPU → you sized it correctly.
• Any CPU percentage → part of the model is on your processor. That is the cliff, not a gentle slope: a model 90% on GPU and 10% on CPU is dramatically slower than one fully resident.
• CONTEXT far lower than you expected → your runner made room by shrinking the window, or never had the VRAM it needed.

If it does not fit

In order of what to try:

1. Drop a quantization level. Q5 → Q4 is usually invisible. Q4 → Q3 usually is not.
2. Shorten the context. A 128k window you do not use is memory you are wasting.
3. Drop a model size. 13B → 7B. Last resort, biggest quality cost.

What not to do: run it half on the CPU and tell yourself it is fine. Fix the sizing or use a smaller model — a fast small model beats a slow big one for everything except bragging.