What is a GGUF file?

GGUF is a file format for distributing models you run locally. If you have downloaded a model to run on your own machine, it was almost certainly a .gguf file.

It was created for the llama.cpp project, and because llama.cpp is the engine underneath Ollama and LM Studio, GGUF is effectively the standard container for local inference.

What it does

One file, everything inside: the weights, the tokenizer, the metadata, the architecture description. No separate config files, no directory of shards. Download one file, point a runner at it, done.

That sounds mundane. It is the entire reason local AI is approachable — the previous state of affairs was a directory of tensors plus config files plus a Python environment, and version mismatches everywhere.

It is also designed for memory mapping: the runner reads weights straight from disk rather than loading the whole file into RAM first. This is why a model can start answering almost immediately rather than after a two-minute load.

Reading the filename

This is the useful part, and nothing tells you it. A GGUF filename looks like:

model-name-7B-Q4_K_M.gguf

Broken down:

Part	Meaning
`7B`	7 billion parameters — the model's size
`Q4`	4-bit quantization — the main memory driver
`K`	Uses the "K-quant" method — better quality per bit than the older scheme
`M`	Size within that class: `S` small, `M` medium, `L` large

So Q4_K_M means 4-bit, K-quant, medium — and it is the most common choice for good reason: it is the usual sweet spot between size and quality.

Rough guide to what the numbers cost you:

Suffix	Roughly	Use when
`Q8_0`	8-bit, near-lossless	You have memory to spare
`Q6_K`	6-bit	Quality matters, memory is fine
`Q5_K_M`	5-bit	A good compromise
`Q4_K_M`	4-bit	The default choice
`Q3_K_M`	3-bit	Desperate
`Q2_K`	2-bit	Usually not worth running

F16 or BF16 in a filename means not quantized at all — full size, and the reason a "small" 7B model can be a 14 GB download.

The one rule

Pick the largest model whose file fits in your memory with room to spare — and leave headroom, because the context window needs memory too, on top of the file size.

A 4-bit 13B model generally beats an 8-bit 7B model at the same memory budget. The number of parameters usually matters more than the precision they are stored at.

What we addedA decoder for GGUF filenames — the Q-numbers and letter suffixes that tell you the memory footprint, which no download page explains.

This article was researched and drafted with AI assistance from the sources listed below, then checked and edited by Fiqhro Dedhen before publication. How we work.