Updated: 2026-07-07

Three years ago, NPUs were a marginal checkbox on a laptop spec sheet. Today they define the real experience when running local models for transcription, summarization, image generation, or code assistance. The leap happened faster than predicted and the 2026 hardware map looks very different from 2024.

Key takeaways

  • An NPU trades versatility for performance per watt: ideal for continuous inference, not training.

  • TOPS metrics mislead on their own; compiler maturity and memory bandwidth determine real performance.

  • Apple, Qualcomm, Intel, and AMD concentrate the four major consumer NPU ecosystems.

  • Continuous inference, models up to 13B parameters, and privacy-sensitive on-device data are the three star use cases.

  • NVIDIA GPUs still dominate training and large models (over 30B).

What an NPU actually is

An NPU (Neural Processing Unit) is an accelerator designed specifically for the operations dominating neural-network inference: matrix-matrix multiplications, convolutions, and activations. Unlike GPUs, which are general-purpose parallel compute accelerators, NPUs trade versatility for much higher performance per watt within their narrow domain. That makes them ideal for continuous or low-latency workloads on devices with limited thermal budget.

The most-published metric is TOPS (tera-operations per second) in INT8 or FP16 precision. It’s useful as order of magnitude but misleading on its own: it captures neither available memory bandwidth nor the efficiency of the compiler translating the model into accelerator instructions. A 40 TOPS chip with a mature compiler beats a 50 TOPS chip with poor software support in practice.

The other factor that matters is unified memory. NPUs sharing memory with CPU and GPU avoid costly copies when the pipeline combines multiple stages, increasingly common because real models are rarely a single isolated neural network.

The four actors dominating 2026

The consumer market has consolidated into four main families, each with distinct characteristics and a clear position against the others.

Apple and its Neural Engine

Apple leads on ecosystem coherence. The M4’s 16-core Neural Engine officially reaches 38 TOPS[1], with unified memory shared with CPU and GPU. Apple hasn’t published its own TOPS figure for the M5; instead it adds a neural accelerator inside every GPU core, so a good share of inference now runs outside the classic Neural Engine. The tool chain (Core ML, MLX, Metal) lets developers move loads between accelerators without rewriting code, and models such as Phi-3, Llama 3.2, Mistral Small, and Gemma 2 run comfortably on a consumer Mac with sub-100ms-per-token latency.

The weakness is still versatility: the Neural Engine handles standard architectures well but is more rigid than an NVIDIA GPU when the model has uncommon operators.

Qualcomm and its Hexagon NPU

Qualcomm has gone from quiet mobile leader to the actor defining local AI on Windows ARM laptops. The first Snapdragon X Elite (2024) shipped with a 45 TOPS Hexagon NPU; the X2 Elite generation, launched at CES in January 2026, pushes that to 80 TOPS[2] with its sixth-generation Hexagon NPU. The AI Engine Direct stack integrates well with ONNX Runtime, DirectML, and the new Windows ML. In practice, a Snapdragon X2 Elite laptop runs 7-13 billion parameter models with better battery life than an equivalent x86 laptop with comparable discrete GPU for continuous workloads.

Qualcomm’s challenge is software: drivers matured during 2025 but inconsistencies still appear with less popular frameworks.

Intel NPU 5 in Core Ultra 300

Intel made a notable leap with the NPU 5 in Core Ultra 300 (code name Panther Lake, unveiled at CES in January 2026): from 11 TOPS in the first Meteor Lake generation (2023) to the current NPU 5’s 50 TOPS[3], with major improvements in bandwidth and the OpenVINO compiler. Intel’s clear bet is that developers shouldn’t have to choose between CPU, integrated GPU, or NPU: the OpenVINO runtime picks the optimal route by model and thermal state.

NPU 5 competes well with Apple’s Neural Engine on standard loads, though it trails on energy efficiency during long continuous inference. For corporate environments with majority Windows fleets, Intel is again a reasonable option.

AMD and its XDNA2 in Ryzen AI

AMD entered the NPU segment later, but the XDNA2 integrated in Ryzen AI 300 (2024) already arrived with 50 TOPS[4] and good ROCm and ONNX support; the Ryzen AI 9 HX 375 variant bumps that to 55 TOPS. The Ryzen AI 400 family (2026) keeps the same XDNA2 architecture but pushes mobile models up to 60 TOPS. Unified memory between CPU, integrated Radeon GPU, and NPU works well for hybrid pipelines, and the software ecosystem has professionalized enough to treat AMD as a viable option. AMD’s strongest point is performance per euro: in midrange laptops, Ryzen AI 350 chips offer inference capacity comparable to pricier solutions, with a clear final-price advantage, the same pattern already seen with ONNX Runtime at the edge in other ARM ecosystems.

Which workloads pay off on NPU

Not all AI workloads are equal. Three types are clearly best on NPU:

Continuous low-latency inference: voice transcription, noise cancellation in calls, real-time camera effects. These run for hours and NPU performance per watt crushes GPU.

Small and medium models (up to 13B parameters in INT4) fitting in device memory. The NPU runs them with low latency without heating the laptop. For local assistants, translation, or short text generation, the experience is qualitatively different from sending every request to the cloud.

Privacy-sensitive inference where data can’t leave the device for legal or contractual reasons. Here the NPU is a direct enabler, not just an optimization.

Where NPUs still lose: training, very large models (over 30B parameters), and workloads with non-standard operators. NVIDIA GPUs or dedicated datacenter accelerators still rule there.

A code example

from transformers import pipeline
import torch

# Backend selected by platform: "coreml" (Apple),
# "qnn" (Qualcomm), "openvino" (Intel), or "rocm" (AMD).
pipe = pipeline(
    "text-generation",
    model="microsoft/Phi-3-mini-4k-instruct",
    device="npu",
    torch_dtype=torch.int8,
)

response = pipe("Summarize the following text...", max_new_tokens=128)

The runtime picks the real route. What used to demand hardware-family-specific code now starts hiding behind reasonable common abstractions.

My reading

NPUs have crossed the line between marginal novelty and decisive component. In 2026 a laptop without a competent NPU is an old laptop, not one without extras. For application developers, the decision is no longer whether to embed local AI but which runtime to use so the code leverages the available accelerator without vendor-specific branches.

What remains open is software consolidation. Teams shipping apps with local AI are making pragmatic decisions: use ONNX Runtime with the matching execution provider, add an Apple-specific Core ML layer, and accept that perfect portability will still take a couple of years. Anyone who has weighed NPUs for developers or compared options in NPU in the PC will recognize the same trade-off between performance per watt and software maturity.

This article is also available in Spanish.

Sources

  1. 38 TOPS
  2. pushes that to 80 TOPS
  3. 50 TOPS
  4. 50 TOPS