A couple of years ago, talking about an NPU in a normal PC was a rarity. Today, if you buy a mid-range laptop, it almost certainly has one. Qualcomm put it in first with the Snapdragon X, Intel followed with Lunar Lake, and AMD closed the loop with Strix Point APUs. Microsoft used the moment to create the Copilot+ PC category, which requires at least 40 TOPS of NPU and has finished pushing the market in that direction.

The question I want to answer here is not whether NPUs exist, but whether they actually change anything for people wanting to run AI models locally. I’ve tested different setups for months, and the answer is more nuanced than the announcements suggest.

Key takeaways

  • Current NPUs are optimised for inference with quantised weights in INT8 or INT4: they are not general-purpose GPUs.

  • A 40 TOPS NPU can run quantised Phi-3 Mini consuming 5-10 W, vs 40-50 W for a laptop GPU doing the same task.

  • The software ecosystem is the weak point: QNN, OpenVINO and ROCm are different runtimes requiring specifically optimised models.

  • Ollama still does not use NPUs in most configurations; it goes through CPU or GPU.

  • Where NPU makes a real difference is in ultrabooks and silent mini PCs: continuous inference without a fan.

What an NPU does well

Current NPUs are optimized for neural-network inference with weights quantized to INT8 or INT4. They are not general-purpose processors or GPU replacements: they are accelerators specialized in low-precision matrix multiplications with minimal power draw.

That is where they win. A 40 TOPS NPU can run a small language model (say, Phi-3 Mini quantized) at a perfectly usable speed while drawing 5 to 10 watts. A laptop GPU would do the same task faster, but consume four or five times more energy, with the fan spinning and the battery visibly dropping.

The second scenario where they shine is computer vision: object detection, segmentation, face recognition on local cameras, real-time video filtering.

Where they don’t reach

It is important not to confuse NPUs with a GPU replacement for heavier workloads. A 7B-parameter model running at Q4 precision will consume 4 to 5 GB of memory. The NPU can handle it, but the available NPU memory, shared with the system in these architectures, limits the practical size. With 16 GB total RAM, running a 7B while working on something else is uncomfortable.

They are also not designed for training, or serious fine-tuning. When announcements talk about "AI on your PC", the assumed scenario is always inference on pre-trained models.

The state of software

This is the point where things are less mature. Having an NPU does not mean any app will use it automatically. Each vendor exposes its NPU via its own runtime:

  • QNN on Qualcomm.

  • OpenVINO on Intel.

  • ROCm/Ryzen AI on AMD.

Interop has improved a lot with ONNX Runtime, which abstracts the three platforms, but the reality is the ecosystem still assumes CPU or GPU and ignores the NPU entirely.

To run an LLM locally, the most practical options today are:

  • LM Studio: has started supporting Qualcomm NPUs via QNN.

  • ONNX Runtime with optimized models: direct but more technical route.

  • Ollama: still does not use NPUs in most setups; goes through CPU or GPU.

When a PC with NPU is worth picking

My practical recommendation: if you buy a new laptop, almost any mid- or high-range machine will ship with an NPU, so what matters is how to weigh it.

If you will be using local AI models routinely (always-on assistants, real-time video processing, audio transcription), the NPU adds value. It extends battery and lets you keep the feature on without noticeable impact.

If your local-AI usage is occasional (load a model, run tests, generate some text), a reasonable GPU is still more versatile.

Where I would pick carefully is in small, silent machines: ultrabooks, mini PCs, tablets. There, the NPU makes a real difference. A Snapdragon X Elite can run a decent Phi-3 without a fan, and no integrated GPU matches that.

Looking a bit ahead

What I think will happen over the next two years is that the ecosystem will level out. Runtimes will converge, popular frameworks will start exposing the NPU as a default option, and small models specifically designed for NPUs will become the natural use case. The transition echoes GPUs for scientific computing fifteen years ago: early on they needed special compilers and rewritten code; today anyone uses them without thinking.

Conclusion

NPUs in consumer PCs are hardware ahead of their software, but that gap is closing. If you already have one in your machine, it is worth knowing it is there and starting to try what it enables. For new purchases in 2025, the NPU is not the deciding factor, except in silent ultrabooks or mini PCs where energy efficiency is critical: there it does tip the balance. Getting to know it today is a reasonable afternoon investment, and it saves you from ignoring, a year from now, an accelerator you already have at home.

Frequently asked questions

Can I run a 7B model on the NPU of a laptop with 16 GB of RAM?

You can, but it is uncomfortable. A 7B-parameter model at Q4 precision takes 4 to 5 GB. Since the NPU shares memory with the system in these architectures, on 16 GB of total RAM running it while you work on something else gets awkward. The NPU's natural ground is small models such as quantized Phi-3 Mini, which a 40 TOPS NPU runs at a perfectly usable speed while drawing 5 to 10 watts.

Does Ollama use my laptop's NPU?

Not yet in most configurations: Ollama goes through CPU or GPU and ignores the NPU. Each vendor exposes it through its own runtime (QNN on Qualcomm, OpenVINO on Intel, ROCm/Ryzen AI on AMD), and ONNX Runtime abstracts the three. But the wider app ecosystem still assumes CPU or GPU. If you want to use it for an LLM today, LM Studio has started supporting Qualcomm NPUs via QNN, and ONNX Runtime with optimized models is the direct but more technical route.

How much power does the NPU save compared with the GPU for local inference?

A lot: a 40 TOPS NPU runs quantized Phi-3 Mini at 5-10 W, versus 40-50 W for a laptop GPU doing the same task, roughly four or five times less energy. The GPU finishes faster, but with the fan spinning and the battery visibly dropping. That saving is what makes the difference in silent ultrabooks, mini PCs and tablets: a Snapdragon X Elite can run a decent Phi-3 without a fan, something no integrated GPU matches.