ONNX

07/19/2026

• What is ONNX
• ONNX (Open Neural Network Exchange) is an open, framework-agnostic format for neural networks
• A model is stored as a computation graph, where nodes are standardized operators (Gemm, Relu, Conv, …) and edges are the tensors flowing between them
• It decouples a model from the framework that trained it: the same .onnx file can be produced by PyTorch, TensorFlow, or scikit-learn, and read by any ONNX-compatible runtime
• An ONNX file is a single format that many frameworks can produce and many runtimes and hardware backends can run, so a model trained once works anywhere without change



flowchart LR
    pt["PyTorch"] --> onnx
    tf["TensorFlow"] --> onnx
    sk["scikit-learn"] --> onnx
    kr["Keras"] --> onnx
    onnx["ONNX (.onnx)"] --> ort["ONNX Runtime"]
    ort --> cpu["CPU"]
    ort --> cuda["CUDA"]
    ort --> coreml["CoreML"]
    ort --> webgpu["WebGPU"]
    ort --> wasm["WASM"]

    classDef hub fill:#212529,stroke:#212529,color:#ffffff;
    classDef engine fill:#495057,stroke:#495057,color:#ffffff;
    classDef browser fill:#d0e2ff,stroke:#9dbef0,color:#1a1a1a;
    class onnx hub
    class ort engine
    class webgpu,wasm browser

ONNX decouples training frameworks from runtimes and hardware


• Exporting a model to ONNX
• torch.onnx.export traces the model once with a dummy input and records the operators it runs into a graph
• A tiny model is enough to see the whole shape of it. Export a 3 → 1 linear layer to one self-contained model.onnx:

  
      import torch
      import torch.nn as nn
  
      model = nn.Linear(3, 1).eval()  # tiny model: 3 numbers in, 1 out
      dummy = torch.rand(1, 3)  # one example input
  
      torch.onnx.export(
          model,
          dummy,
          "model.onnx",
          input_names=["input"],
          output_names=["output"],
          dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
          opset_version=17,
      )
  

• The key arguments:
  • input_names / output_names name the graph's inputs and outputs (input → output)
  • dynamic_axes marks which axes may vary at runtime; here axis 0, so one exported model accepts any batch size
  • opset_version picks which version of the ONNX operator set to target
• The result is a single .onnx file holding both the graph and the weights. No Python or PyTorch is needed to run it


• Running it with ONNX Runtime
• ONNX Runtime (ORT) is the engine that loads an .onnx graph and executes it. Running the file we just exported takes a few lines:

  
      import numpy as np
      import onnxruntime as ort
  
      session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
  
      x = np.random.rand(4, 3).astype("float32")  # a batch of 4
      out = session.run(None, {"input": x})
      print(out[0].shape)  # (4, 1)
  

• Execution providers are pluggable backends, so the same file runs on CPU, CUDA, TensorRT, CoreML, WebGPU, or WASM. The providers=[...] argument above chose the CPU one; swap it and the graph stays unchanged


• ONNX Runtime in the browser
• onnxruntime-web is the WASM / WebGPU build of ORT that runs inside a browser, with no server and no Python
• The same model.onnx loads client-side, with a graceful fallback that tries WebGPU and drops to WASM if the browser lacks it:

  
      import { InferenceSession, Tensor } from "onnxruntime-web/webgpu";
  
      const session = await InferenceSession
          .create("model.onnx", { executionProviders: ["webgpu"] })
          .catch(() => InferenceSession.create("model.onnx", { executionProviders: ["wasm"] }));
  
      const x = new Tensor("float32", new Float32Array(4 * 3), [4, 3]);
      const out = await session.run({ input: x });
  

• The WASM path is plain WebAssembly: if the page can't send the COOP/COEP headers that SharedArrayBuffer threads need (GitHub Pages, for example), ORT falls back to running single-threaded


• A performance problem with dynamic shapes
• The flexibility from dynamic_axes is not free at runtime. GPU backends compile a kernel pipeline per input shape, so if the batch size changes on every call, ORT recompiles every run
• The fix is to pin the shape: tell ORT the axis is fixed, so the pipeline compiles once and is reused across calls:

  
      // fix the "batch" axis so the WebGPU pipeline compiles ONCE, not per call
      const opts = { executionProviders: ["webgpu"], freeDimensionOverrides: { batch: 4 } };
      const session = await InferenceSession.create("model.onnx", opts);
  

• A dynamic axis is convenient at export time, but the runtime may pay for it with recompilation. Fixing the shape at inference time is a standard performance lever



%%{init: {"flowchart": {"rankSpacing": 20, "nodeSpacing": 30}}}%%
flowchart TB
    subgraph fixed[" "]
        direction LR
        f1["batch 4"] --> fc["compile"] --> fr1["run"]
        f2["batch 4"] --> fr2["run"]
        f3["batch 4"] --> fr3["run"]
        f2 ~~~ fc2[" "] ~~~ fr2
        f3 ~~~ fc3[" "] ~~~ fr3
    end
    vs["vs."]
    subgraph dynamic[" "]
        direction LR
        d1["batch 4"] --> dc1["compile"] --> dr1["run"]
        d2["batch 8"] --> dc2["compile"] --> dr2["run"]
        d3["batch 16"] --> dc3["compile"] --> dr3["run"]
    end
    fixed ~~~ vs ~~~ dynamic

    classDef compile fill:#f8d7da,stroke:#d98a92,color:#1a1a1a;
    classDef blank fill:transparent,stroke:transparent,color:transparent;
    class fc,dc1,dc2,dc3 compile
    class fc2,fc3 blank
    style fixed fill:transparent,stroke:transparent
    style dynamic fill:transparent,stroke:transparent
    style vs fill:transparent,stroke:transparent

The top (fixed shape) compiles once and reuses it. The bottom (dynamic shape) recompiles on every call.

References:

• ONNX (Open Neural Network Exchange)
• ONNX Runtime
• ONNX Runtime execution providers
• torch.onnx (PyTorch documentation)
• onnxruntime-web

---