Updated: 2026-07-07

Deploying a machine-learning model outside the notebook where it was trained is usually where the fantasy breaks: you train in PyTorch on a cloud GPU and suddenly need to serve inference on a Linux server, inside an iOS app, on an ARM industrial gateway, and in a customer’s browser tab. Each destination brings its own runtime. ONNX Runtime[1] is what most teams arrive at when that pain becomes chronic.

Key takeaways

  • ONNX Runtime turns the ONNX format into a usable tool: export once from PyTorch or TensorFlow and run almost anywhere with the same artifact.

  • Execution Providers (EPs) separate the graph from hardware; the same code runs on a datacenter GPU in development and an edge CPU in production.

  • Dynamic quantization is the cheap entry: one line, no calibration, 4× size reduction for typical CNNs.

  • The real state of NPU EPs in Q1 2024 lags behind the commercial narrative.

  • If deployment is exclusively large NVIDIA GPU, TensorRT or vLLM will probably win.

What ONNX actually solves

The problem is not purely technical, it’s organisational. An ML team that trains in PyTorch and a mobile team that integrates on iOS speak different languages. Without a bridge format, each new target adds weeks of re-engineering: convert the graph, validate outputs match within tolerance, rediscover which operators aren’t supported.

ONNX cuts that knot by proposing an open intermediate format: a computational graph with standard operators versioned by opset. ONNX Runtime is the reference implementation: a single .onnx artifact that works for server, mobile, browser and edge without duplicating tooling.

If a team lives entirely inside one ecosystem, the native alternative usually performs better. TensorRT gets more out of an NVIDIA GPU; Core ML squeezes an A17 Pro harder. But as soon as a second target shows up, the cost of maintaining two or three native runtimes outweighs whatever was lost by picking the common denominator.

Export: the step that looks easy

All the value of ONNX Runtime depends on export working correctly. In PyTorch, the critical element is declaring which axes are dynamic:

import torch

dummy_input = torch.randn(1, 3, 224, 224)

torch.onnx.export(
    model,
    dummy_input,
    "model.onnx",
    input_names=["input"],
    output_names=["output"],
    dynamic_axes={
        "input": {0: "batch_size"},
        "output": {0: "batch_size"},
    },
    opset_version=17,
)

Without dynamic_axes, the model is frozen to batch size 1 and breaks in production with different batch sizes. opset_version=17 is a good middle ground: modern enough for recent models and old enough that every EP understands it.

The non-negotiable step after export is validation: feed the same tensor to both the PyTorch model and the ONNX Runtime session and compare with np.allclose at a conservative tolerance. Typical sources of mismatch: custom operators translated incorrectly, and float32/float16 differences.

Execution Providers: Where the Performance Lives

The architecture separates the graph from the hardware through Execution Providers. The default EP is CPU (optimised with MLAS), but the runtime’s real argument is the list stacked on top of it:

  • CUDA and TensorRT for NVIDIA.

  • OpenVINO for Intel.

  • DirectML for Windows with any GPU.

  • ROCm for AMD on Linux.

  • Core ML for Apple Silicon.

  • NNAPI for Android.

  • QNN for Snapdragon.

  • WebGPU and WebAssembly for the browser.

Configuration is a prioritised list: the runtime tries the first EP, falls to the next if hardware isn’t available, and lands on CPU as the safety net. The same Python code runs inference on the datacenter GPU in development and on the edge CPU in production without touching a line.

Quantization and Graph Optimisation

At load time, ONNX Runtime applies automatic passes (operator fusion, constant folding, dead-node elimination) with no intervention needed. The big jump in size and latency comes from quantization:

  • Dynamic: one line, no calibration, acceptable for most CNNs. 4× size reduction.

  • Static: with a representative calibration dataset. Needed for the full 2-4× latency gain.

On large models like Whisper or BERT, INT8 is often the difference between running on a decent phone and not running at all.

Browser, Mobile, and the Real Edge Case

onnxruntime-web runs ONNX models in the browser using WebGPU when available and WebAssembly as fallback. For image classification, detection, or a Whisper-tiny transcribing on the client, the need for an inference service and its GPU bill disappears.

On mobile, ONNX Runtime Mobile produces .aar for Android and Swift/Objective-C frameworks for iOS. On embedded edge (Jetson, Raspberry Pi, ARM industrial gateways) the argument is portability: iterate on a workstation with CUDA, validate on a laptop with CPU, and deploy on Jetson Orin with the TensorRT EP without rewriting anything.

The ability to serve quantized models on heterogeneous devices is where ONNX Runtime pays off the most in projects that mix servers, mobile, and industrial edge.

Conclusion

ONNX Runtime isn’t the fastest engine on any specific platform and almost no single benchmark will crown it champion. It doesn’t pretend to. Its proposition is to absorb the complexity of heterogeneity: one artifact, one API, many targets, and enough performance margin on each to make portability pay. The honest caveat in Q1 2024 is the state of NPUs: commercial narrative runs ahead of operational reality. For everything else (CPU, consumer GPU, browser, classic mobile) it’s already the sensible choice.

Sources

  1. ONNX Runtime