What is an NPU?

An NPU — neural processing unit — is a chip built for one job: running the maths behind AI models quickly and, above all, using very little power. Recent laptops and phones ship with one, and the marketing tends to lead with a single figure: TOPS. That number is worth understanding, mostly so you know how little it tells you.

What it actually is

A CPU is a generalist. A GPU is built for the massively parallel maths of graphics, which happens to suit AI too. An NPU is narrower still: hardware dedicated to the "deep learning math operations that make up AI models", in Microsoft's words. Its advantage is not raw speed — it is efficiency. It does that maths using far less power than a CPU or GPU, which is why it turns up in battery-powered devices and why phones and thin laptops can now run some AI features without the fan spinning up.

That efficiency comes with a constraint worth knowing: NPUs are tuned for low-precision integer maths, often 8-bit, so a model usually has to be quantized down before it can run on one at all.

What "TOPS" means

TOPS is trillions of operations per second — a throughput rating. Microsoft's bar for its "Copilot+ PC" badge is an NPU that can do 40+ TOPS, and chip vendors — Qualcomm's Snapdragon X, Intel's Core Ultra, AMD's Ryzen AI — now print the figure on the box.

Here is the catch. TOPS is a theoretical peak, like a car's top speed: it tells you what the chip can do under ideal conditions, not what it will do on your workload. And it tells you little about the thing that usually decides whether a device can run, say, a local chatbot — memory.

TOPS is not the number that matters

For the always-on features NPUs were designed for — live captions, background blur, on-device translation, small assistant tasks — TOPS is a reasonable rough guide. For running a general-purpose local LLM, it is close to irrelevant. Those models are limited by how much memory you have and how fast that memory is, not by how many operations a chip can theoretically do.

The number on the box	What actually determines local-AI usefulness
TOPS — theoretical peak operations per second	Memory capacity — the model has to fit in memory to run; too big and it will not load
Higher looks faster	Memory bandwidth — how fast weights move; this, more than TOPS, tends to set generation speed
One figure for every task	Model size — a bigger quantized model needs more room; the GGUF filename tells you how much
Marketed as "AI performance"	For a large model the NPU often is not even the chip doing the work — it runs on the GPU

So what is the NPU for?

Real things, just not the thing the number implies. It is what lets your phone transcribe a voice note, blur your video-call background, or run a small on-device assistant without draining the battery or shipping your data to a server. On a phone, an NPU is a large part of why a small model can run locally at all.

What it is not is a shortcut to running a large language model on your laptop. If that is your goal, the questions to ask are about memory — how much you need — not about TOPS.

What we addedA table separating TOPS — the marketing number — from what actually determines whether a device can run local AI: memory capacity, memory bandwidth, and model size.

This article was researched and drafted with AI assistance from the sources listed below, then checked and edited by Fiqhro Dedhen before publication. How we work.