What is quantization?

A model's weights are numbers. Quantization stores those numbers with less precision — 8 bits instead of 16, or 4 instead of 8. Fewer bits per weight means a smaller file and less memory, which is the difference between a model running on your machine and not running at all.

The trade is accuracy. The useful surprise is how little accuracy you lose.

Why it matters if you run models locally

Memory is the binding constraint on local inference. A rough guide to what a model needs, before context and overhead:

Precision	Bytes per weight	A 7B model needs	A 70B model needs
FP16 (unquantized)	2	~14 GB	~140 GB
8-bit	1	~7 GB	~70 GB
5-bit	~0.65	~4.5 GB	~45 GB
4-bit	~0.5	~3.5 GB	~35 GB

These are approximations — real files vary with architecture and method — but the shape is what matters. At FP16 a 70B model needs multiple data-centre GPUs. At 4-bit it fits on a single high-end consumer card or a Mac with enough unified memory. That is the whole story of local AI in one table.

Add headroom on top: the context window consumes memory too, and it grows with conversation length.

What you give up

Less than people expect, and it is not linear:

8-bit — quality loss is generally negligible. If it fits, take it.
5-bit and 6-bit — small, usually unnoticeable degradation. The common sweet spot.
4-bit — measurable but often acceptable. This is where most local setups land.
3-bit and below — degradation becomes obvious. Usually a false economy.

The rule of thumb that holds up well: a larger model quantized harder usually beats a smaller model at full precision, for the same memory budget. A 4-bit 13B model is typically a better deal than an 8-bit 7B one.

Where it goes wrong

Quantization damage is not evenly distributed. Tasks needing precision — long arithmetic, tight code generation, careful instruction-following — degrade first and most. Casual conversation degrades last. So a heavily quantized model can feel fine in a chat and quietly fail at the thing you actually needed.

Test on your real task, not on vibes.

In practice

If you're using a local runner, quantization is already happening — it's what the letters and numbers in a model filename mean. You mostly just need to pick a file that fits in your memory with room to spare, and prefer the biggest model that does.

What we addedA memory-requirement table by precision level, and a straight answer on which quantization levels are worth using.

This article was researched and drafted with AI assistance from the sources listed below, then checked and edited by Fiqhro Dedhen before publication. How we work.