NPUs for developers: what you can actually do today
Updated: 2026-07-07
For a couple of years, the NPU acronym was mostly a label on laptop boxes and a checked box in a processor spec sheet. In 2025 that has changed enough to warrant an honest review: what hardware is available, what tools let you use it from real code, which workloads actually pay off, and which are still better left on CPU or GPU. The state of things is not uniform and not finished, but there is enough here for a developer to decide with some confidence whether it is worth the time to integrate an NPU into a real product.
Key takeaways
-
The three dominant laptop families are Qualcomm Snapdragon X (Elite and Plus), Apple Silicon from M1 onward, and AMD Ryzen AI 300 with XDNA.
-
TOPS are an easy number to compare but a misleading one: what matters in practice is the combination of raw capacity, supported precision, memory bandwidth, and the software stack available.
-
The most useful common denominator for developers wanting to cover multiple platforms is ONNX Runtime with vendor-specific execution providers.
-
Best-solved use cases today are lightweight vision, audio transcription, and small language models with INT4 quantization.
-
Running a model on NPU is often slower than on integrated GPU for the first invocation because of loading and compilation cost; the benefit shows up in repeated runs.
What’s on the market
The three dominant laptop families are Qualcomm Snapdragon X (Elite and Plus), Apple Silicon from M1 onward (I covered the move from M3 to M4 in more detail earlier), and AMD Ryzen AI 300 with XDNA. Intel came later with the Core Ultra Meteor Lake and Lunar Lake chips, which include their own NPU but with a less mature software ecosystem. The numbers vendors advertise revolve around TOPS: 45 on Snapdragon X Elite[1], 38 on Apple M4[2], 50 on Ryzen AI 300[3], and 48 on Lunar Lake[4].
TOPS are an easy number to compare but a misleading one. What matters in practice is the combination of raw capacity, supported precision, memory bandwidth, and the software stack available to actually reach the silicon. A chip with 45 theoretical TOPS and immature tooling delivers less real inference than one with 30 TOPS and a polished toolchain.
The toolchain: ONNX Runtime as common denominator
The element that made it realistic to talk about NPUs for developers is ONNX Runtime[5] (I wrote about ONNX Runtime at the edge in more depth) with vendor-specific execution providers:
-
For Qualcomm there is QNN EP.
-
For Apple, CoreML EP.
-
For AMD, the Vitis AI EP.
-
For Intel, the OpenVINO EP.
All of them follow the same pattern: take a model in ONNX format and dispatch part of the graph to the NPU, leaving the rest on CPU or GPU.
Every vendor also has its own dedicated toolchain:
-
Apple offers Core ML with its
coremltoolscompiler, which converts models from PyTorch or ONNX. -
AMD has Ryzen AI Software, built on a Vitis AI flow that compiles models quantized to INT8.
-
Qualcomm provides the AI Engine Direct SDK with conversion utilities to its own QNN binary format.
-
Intel pushes OpenVINO, which besides its NPU also supports CPU and integrated GPU through the same API.
The practical decision for a developer who wants to cover multiple platforms is to start with ONNX Runtime. A model well exported to ONNX can run on CPU, GPU, and the four major NPUs with minimal changes to the inference code. Quantization to INT8 or INT4 is almost always required: most NPUs are built around integer arithmetic, and getting full value out of them means reducing model precision at export time, not at load time.
What you can do today
The best-solved use case today is lightweight vision inference. Object detection, image classification, segmentation, and face recognition all run well on any current NPU with tens-of-milliseconds latency and meaningfully lower power draw than on integrated GPU. For desktop applications that process live camera video, the NPU is the natural choice today.
The second mature case is audio transcription. Whisper in its small and medium variants runs reasonably well on NPU after proper quantization. Apple has very polished Whisper support on the Neural Engine through Core ML; the other vendors have been catching up through 2025 with varying levels of quality.
The third case, more recent and more ambitious, is small language models. Phi-3 Mini, Llama 3.2 1B and 3B, and Qwen 2.5 in the few-billion-parameter range, quantized to INT4, already run on current NPUs at a tokens-per-second rate that starts to be useful for summarization, text correction, or local assistants. This is not the territory where a laptop NPU competes with a datacenter GPU; it is the territory where it competes with running the same model on CPU, and there the NPU usually wins clearly on both latency and energy.
This kind of local inference for small models connects directly with the patterns I covered in AI agent observability: agents that run part of their pipeline locally on the NPU need their own instrumentation to understand which subsystem is doing which part of the work.
Where it doesn’t pay off yet
Large models remain GPU or abundant-memory CPU territory. A 13-billion-parameter model does not fit in the memory reachable by a laptop NPU, or it fits only under heavy quantization with degraded quality. The same applies to large diffusion models, to training workloads (no consumer NPU trains today), and to loads with complex control flow that do not compile well to the static graph an NPU expects.
It also doesn’t pay off when inference runs server-side and the client only makes an HTTP request. The NPU’s territory is local inference, and if your architecture does not involve local execution, the NPU is a non-problem.
A detail that surprises people approaching this for the first time is that running a model on NPU is often slower than on integrated GPU for the first invocation, because of loading and compilation cost. The benefit shows up in repeated runs or in long-running scenarios, where the energy efficiency offsets the initial latency.
Minimal ONNX Runtime example
For a developer who wants to try this today, the short path is to export a model to ONNX from PyTorch, quantize it, and load it with the matching execution provider:
import onnxruntime as ort
import numpy as np
# On a Snapdragon X Elite laptop
providers = [
("QNNExecutionProvider", {"backend_path": "QnnHtp.dll"}),
"CPUExecutionProvider",
]
session = ort.InferenceSession("quantized_model.onnx", providers=providers)
input_data = np.random.randn(1, 3, 224, 224).astype(np.float32)
output = session.run(None, {"input": input_data})On Apple switch to CoreMLExecutionProvider, on AMD to VitisAIExecutionProvider, on Intel to OpenVINOExecutionProvider. The idea is that the same model and almost the same code run on all four, and that if something fails on the NPU the runtime falls back to CPU automatically. Reality has more edge cases than that, but the abstraction is useful as a starting point.
My take
After following this space for two years, laptop NPUs are today a real tool, not a magic fix. For specific cases (lightweight vision, audio, small language models) they pay off clearly in latency and energy. For large cases, they don’t. The toolchain has matured enough that a developer with inference experience can integrate an NPU into a product in weeks, not months, as long as they accept the limits that quantization imposes.
The most frequent blind spot I see in teams is assuming TOPS numbers translate directly into performance. They don’t. What translates into performance is the match between the model, the quantization, the supported operator graph, and the available memory bandwidth.
The direction of travel, though, is clear: every laptop generation lowers the cost of local inference, small models keep gaining capability, and the NPU is settling in as the natural place to run them. What was marketing is becoming infrastructure; it’s time to learn to use it.
Conclusion
Targeting the NPU in 2026 is a reasonable bet for products that need to work offline or with low perceived latency. The correct evaluation is always on the target hardware, with your own model, and with real data; vendor benchmarks are starting points, not decisions.
This article is also available in Spanish: NPU para desarrolladores: qué se puede hacer ya.
