latentspace.

ONNX


  • What is ONNX
    • ONNX (Open Neural Network Exchange) is an open, framework-agnostic format for neural networks
    • A model is stored as a computation graph, where nodes are standardized operators (Gemm, Relu, Conv, …) and edges are the tensors flowing between them
    • It decouples a model from the framework that trained it: the same .onnx file can be produced by PyTorch, TensorFlow, or scikit-learn, and read by any ONNX-compatible runtime
    • An ONNX file is a single format that many frameworks can produce and many runtimes and hardware backends can run, so a model trained once works anywhere without change

    • flowchart LR
          pt["PyTorch"] --> onnx
          tf["TensorFlow"] --> onnx
          sk["scikit-learn"] --> onnx
          kr["Keras"] --> onnx
          onnx["ONNX (.onnx)"] --> ort["ONNX Runtime"]
          ort --> cpu["CPU"]
          ort --> cuda["CUDA"]
          ort --> coreml["CoreML"]
          ort --> webgpu["WebGPU"]
          ort --> wasm["WASM"]
      
          classDef hub fill:#212529,stroke:#212529,color:#ffffff;
          classDef engine fill:#495057,stroke:#495057,color:#ffffff;
          classDef browser fill:#d0e2ff,stroke:#9dbef0,color:#1a1a1a;
          class onnx hub
          class ort engine
          class webgpu,wasm browser
      
      ONNX decouples training frameworks from runtimes and hardware

  • Exporting a model to ONNX
    • torch.onnx.export traces the model once with a dummy input and records the operators it runs into a graph
    • A tiny model is enough to see the whole shape of it. Export a 3 → 1 linear layer to one self-contained model.onnx:
      
          import torch
          import torch.nn as nn
      
          model = nn.Linear(3, 1).eval()  # tiny model: 3 numbers in, 1 out
          dummy = torch.rand(1, 3)  # one example input
      
          torch.onnx.export(
              model,
              dummy,
              "model.onnx",
              input_names=["input"],
              output_names=["output"],
              dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
              opset_version=17,
          )
      
    • The key arguments:
      • input_names / output_names name the graph's inputs and outputs (input → output)
      • dynamic_axes marks which axes may vary at runtime; here axis 0, so one exported model accepts any batch size
      • opset_version picks which version of the ONNX operator set to target
    • The result is a single .onnx file holding both the graph and the weights. No Python or PyTorch is needed to run it

  • Running it with ONNX Runtime
    • ONNX Runtime (ORT) is the engine that loads an .onnx graph and executes it. Running the file we just exported takes a few lines:
      
          import numpy as np
          import onnxruntime as ort
      
          session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
      
          x = np.random.rand(4, 3).astype("float32")  # a batch of 4
          out = session.run(None, {"input": x})
          print(out[0].shape)  # (4, 1)
      
    • Execution providers are pluggable backends, so the same file runs on CPU, CUDA, TensorRT, CoreML, WebGPU, or WASM. The providers=[...] argument above chose the CPU one; swap it and the graph stays unchanged

  • ONNX Runtime in the browser
    • onnxruntime-web is the WASM / WebGPU build of ORT that runs inside a browser, with no server and no Python
    • The same model.onnx loads client-side, with a graceful fallback that tries WebGPU and drops to WASM if the browser lacks it:
      
          import { InferenceSession, Tensor } from "onnxruntime-web/webgpu";
      
          const session = await InferenceSession
              .create("model.onnx", { executionProviders: ["webgpu"] })
              .catch(() => InferenceSession.create("model.onnx", { executionProviders: ["wasm"] }));
      
          const x = new Tensor("float32", new Float32Array(4 * 3), [4, 3]);
          const out = await session.run({ input: x });
      
    • The WASM path is plain WebAssembly: if the page can't send the COOP/COEP headers that SharedArrayBuffer threads need (GitHub Pages, for example), ORT falls back to running single-threaded

  • A performance problem with dynamic shapes
    • The flexibility from dynamic_axes is not free at runtime. GPU backends compile a kernel pipeline per input shape, so if the batch size changes on every call, ORT recompiles every run
    • The fix is to pin the shape: tell ORT the axis is fixed, so the pipeline compiles once and is reused across calls:
      
          // fix the "batch" axis so the WebGPU pipeline compiles ONCE, not per call
          const opts = { executionProviders: ["webgpu"], freeDimensionOverrides: { batch: 4 } };
          const session = await InferenceSession.create("model.onnx", opts);
      
    • A dynamic axis is convenient at export time, but the runtime may pay for it with recompilation. Fixing the shape at inference time is a standard performance lever

    • %%{init: {"flowchart": {"rankSpacing": 20, "nodeSpacing": 30}}}%%
      flowchart TB
          subgraph fixed[" "]
              direction LR
              f1["batch 4"] --> fc["compile"] --> fr1["run"]
              f2["batch 4"] --> fr2["run"]
              f3["batch 4"] --> fr3["run"]
              f2 ~~~ fc2[" "] ~~~ fr2
              f3 ~~~ fc3[" "] ~~~ fr3
          end
          vs["vs."]
          subgraph dynamic[" "]
              direction LR
              d1["batch 4"] --> dc1["compile"] --> dr1["run"]
              d2["batch 8"] --> dc2["compile"] --> dr2["run"]
              d3["batch 16"] --> dc3["compile"] --> dr3["run"]
          end
          fixed ~~~ vs ~~~ dynamic
      
          classDef compile fill:#f8d7da,stroke:#d98a92,color:#1a1a1a;
          classDef blank fill:transparent,stroke:transparent,color:transparent;
          class fc,dc1,dc2,dc3 compile
          class fc2,fc3 blank
          style fixed fill:transparent,stroke:transparent
          style dynamic fill:transparent,stroke:transparent
          style vs fill:transparent,stroke:transparent
      
      The top (fixed shape) compiles once and reuses it. The bottom (dynamic shape) recompiles on every call.



References: