ONNX
- What is ONNX
- ONNX (Open Neural Network Exchange) is an open, framework-agnostic format for neural networks
-
A model is stored as a computation graph, where nodes are standardized operators (
Gemm,Relu,Conv, …) and edges are the tensors flowing between them -
It decouples a model from the framework that trained it: the same
.onnxfile can be produced by PyTorch, TensorFlow, or scikit-learn, and read by any ONNX-compatible runtime - An ONNX file is a single format that many frameworks can produce and many runtimes and hardware backends can run, so a model trained once works anywhere without change
flowchart LR
pt["PyTorch"] --> onnx
tf["TensorFlow"] --> onnx
sk["scikit-learn"] --> onnx
kr["Keras"] --> onnx
onnx["ONNX (.onnx)"] --> ort["ONNX Runtime"]
ort --> cpu["CPU"]
ort --> cuda["CUDA"]
ort --> coreml["CoreML"]
ort --> webgpu["WebGPU"]
ort --> wasm["WASM"]
classDef hub fill:#212529,stroke:#212529,color:#ffffff;
classDef engine fill:#495057,stroke:#495057,color:#ffffff;
classDef browser fill:#d0e2ff,stroke:#9dbef0,color:#1a1a1a;
class onnx hub
class ort engine
class webgpu,wasm browser
-
torch.onnx.exporttraces the model once with a dummy input and records the operators it runs into a graph -
A tiny model is enough to see the whole shape of it. Export a
3 → 1linear layer to one self-containedmodel.onnx:import torch import torch.nn as nn model = nn.Linear(3, 1).eval() # tiny model: 3 numbers in, 1 out dummy = torch.rand(1, 3) # one example input torch.onnx.export( model, dummy, "model.onnx", input_names=["input"], output_names=["output"], dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}}, opset_version=17, ) -
The key arguments:
input_names/output_namesname the graph's inputs and outputs (input→output)dynamic_axesmarks which axes may vary at runtime; here axis0, so one exported model accepts any batch sizeopset_versionpicks which version of the ONNX operator set to target
-
The result is a single
.onnxfile holding both the graph and the weights. No Python or PyTorch is needed to run it
-
ONNX Runtime (ORT) is the engine that loads an
.onnxgraph and executes it. Running the file we just exported takes a few lines:import numpy as np import onnxruntime as ort session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"]) x = np.random.rand(4, 3).astype("float32") # a batch of 4 out = session.run(None, {"input": x}) print(out[0].shape) # (4, 1) -
Execution providers are pluggable backends, so the same file runs on CPU, CUDA, TensorRT, CoreML, WebGPU, or WASM. The
providers=[...]argument above chose the CPU one; swap it and the graph stays unchanged
- onnxruntime-web is the WASM / WebGPU build of ORT that runs inside a browser, with no server and no Python
-
The same
model.onnxloads client-side, with a graceful fallback that tries WebGPU and drops to WASM if the browser lacks it:import { InferenceSession, Tensor } from "onnxruntime-web/webgpu"; const session = await InferenceSession .create("model.onnx", { executionProviders: ["webgpu"] }) .catch(() => InferenceSession.create("model.onnx", { executionProviders: ["wasm"] })); const x = new Tensor("float32", new Float32Array(4 * 3), [4, 3]); const out = await session.run({ input: x }); -
The WASM path is plain WebAssembly: if the page can't send the COOP/COEP headers that
SharedArrayBufferthreads need (GitHub Pages, for example), ORT falls back to running single-threaded
-
The flexibility from
dynamic_axesis not free at runtime. GPU backends compile a kernel pipeline per input shape, so if the batch size changes on every call, ORT recompiles every run -
The fix is to pin the shape: tell ORT the axis is fixed, so the pipeline compiles once and is reused across calls:
// fix the "batch" axis so the WebGPU pipeline compiles ONCE, not per call const opts = { executionProviders: ["webgpu"], freeDimensionOverrides: { batch: 4 } }; const session = await InferenceSession.create("model.onnx", opts); - A dynamic axis is convenient at export time, but the runtime may pay for it with recompilation. Fixing the shape at inference time is a standard performance lever
%%{init: {"flowchart": {"rankSpacing": 20, "nodeSpacing": 30}}}%%
flowchart TB
subgraph fixed[" "]
direction LR
f1["batch 4"] --> fc["compile"] --> fr1["run"]
f2["batch 4"] --> fr2["run"]
f3["batch 4"] --> fr3["run"]
f2 ~~~ fc2[" "] ~~~ fr2
f3 ~~~ fc3[" "] ~~~ fr3
end
vs["vs."]
subgraph dynamic[" "]
direction LR
d1["batch 4"] --> dc1["compile"] --> dr1["run"]
d2["batch 8"] --> dc2["compile"] --> dr2["run"]
d3["batch 16"] --> dc3["compile"] --> dr3["run"]
end
fixed ~~~ vs ~~~ dynamic
classDef compile fill:#f8d7da,stroke:#d98a92,color:#1a1a1a;
classDef blank fill:transparent,stroke:transparent,color:transparent;
class fc,dc1,dc2,dc3 compile
class fc2,fc3 blank
style fixed fill:transparent,stroke:transparent
style dynamic fill:transparent,stroke:transparent
style vs fill:transparent,stroke:transparent
References:
- ONNX (Open Neural Network Exchange)
- ONNX Runtime
- ONNX Runtime execution providers
- torch.onnx (PyTorch documentation)
- onnxruntime-web