From 173744afe4f81a86e2268d6525e3b89880d77052 Mon Sep 17 00:00:00 2001 From: Reid Morrison Date: Thu, 29 Oct 2026 12:05:25 -0400 Subject: [PATCH] Add AGENTS.md, architecture docs, and a review TODO with roadmap Adds maintainer and agent documentation from a full review of main (55dc3de) on Elixir 1.20.4 / OTP 29.1.2 / Rust 1.99.1: - AGENTS.md: setup, commands, gotchas, design rules, recipes - docs/: architecture, Elixir and native layers, build and packaging, execution providers, testing and CI, and an upgrade guide for Nx 1.0 / Rustler 0.39 / ort 2.0.0-rc.15 (ONNX Runtime 1.31) - TODO.md: prioritized findings, issue and PR triage, phased roadmap Docs only; no code changes. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 165 +++++++++++ TODO.md | 546 ++++++++++++++++++++++++++++++++++++ docs/README.md | 62 ++++ docs/architecture.md | 274 ++++++++++++++++++ docs/build-and-packaging.md | 236 ++++++++++++++++ docs/elixir-layer.md | 159 +++++++++++ docs/execution-providers.md | 123 ++++++++ docs/native-layer.md | 153 ++++++++++ docs/testing-and-ci.md | 139 +++++++++ docs/upgrading.md | 378 +++++++++++++++++++++++++ 10 files changed, 2235 insertions(+) create mode 100644 AGENTS.md create mode 100644 TODO.md create mode 100644 docs/README.md create mode 100644 docs/architecture.md create mode 100644 docs/build-and-packaging.md create mode 100644 docs/elixir-layer.md create mode 100644 docs/execution-providers.md create mode 100644 docs/native-layer.md create mode 100644 docs/testing-and-ci.md create mode 100644 docs/upgrading.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..b84f94b --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,165 @@ +# AGENTS.md + +Instructions for AI coding agents (and humans) working on Ortex. Read this file first, then +[`docs/README.md`](docs/README.md). Known problems and planned work are in [`TODO.md`](TODO.md). + +## What Ortex is + +Ortex is an Elixir library that runs ONNX models through ONNX Runtime. Elixir code +(`lib/`) talks to a Rust NIF (`native/ortex`, built with Rustler) that uses the `ort` crate, which +links ONNX Runtime (a C++ library downloaded at build time). Ortex provides `Ortex.load/3`, +`Ortex.run/2`, a storage-only `Nx.Backend` (`Ortex.Backend`) and an `Nx.Serving` module +(`Ortex.Serving`). + +## Status snapshot (2026-10-29) + +* `main` is `55dc3de`: Hex release `0.1.10` (2024-11-19) plus the Elixir 1.19 `Inspect` fix (PR #48). +* Pins: `ort 2.0.0-rc.8` (ONNX Runtime 1.19.2, max ONNX opset 21), Rustler crate `0.29`, Hex + `{:rustler, "~> 0.27"}`, `{:nx, "~> 0.6"}` (excludes Nx 1.0), `ndarray 0.16`, `edition = "2018"`. +* Verified: `main` compiles and all 117 tests pass on Elixir 1.20.4 / OTP 29.1.2 / Rust 1.99.1, and + also with Nx 1.0.0 + Hex Rustler 0.38 (no code changes). Hex `0.1.10` itself predates PR #48, so + `inspect(model)` is broken for Hex users on Elixir 1.19+ until the next release. +* Time bomb: Rust 1.100 (about 2026-11-12) breaks every dynamically linked build (CUDA/TensorRT, + including this repo's Linux CI) because `ort-sys` rc.8's `copy-dylibs` cannot handle cargo's new + build-dir layout. Verified on 1.100.0-beta.7. See `TODO.md` BUILD-10. +* Upstream today: `ort 2.0.0-rc.15` (still a release candidate), ONNX Runtime 1.31.1, Nx 1.0.0, + Rustler 0.38 (Hex) / 0.39 (crate), rustler_precompiled 0.10.1. +* Open PRs touch most files: #47 (modernise + `ort` rc.11), #49 (precompiled NIFs), #46 (`ort` rc.10), + #50, #51, #52 (session options). Read `TODO.md` section "Pull requests" before starting work + that overlaps them. The maintainer asked for #47 to be split into "modernisation" and "ort upgrade". + +## Repository map + +```text +lib/ortex.ex Public facade: load/3, run/2 (delegates to Ortex.Model) +lib/ortex/model.ex %Ortex.Model{reference}: load, run, Inspect implementation +lib/ortex/backend.ex Ortex.Backend: storage-only Nx.Backend (9 callbacks, rest raise) +lib/ortex/serving.ex Ortex.Serving: Nx.Serving behaviour +lib/ortex/native.ex NIF stubs; builds the Rust crate at compile time; loads the NIF +lib/ortex/util.ex copy_ort_libs/0: copies libonnxruntime* next to the NIF +native/ortex/ Rust crate (cdylib "ortex") + src/lib.rs NIF table (rustler::init!), term decoding, scheduler choice + src/model.rs OrtexModel { session }: init, show, run + src/tensor.rs OrtexTensor enum (one ndarray variant per dtype), conversions + src/utils.rs binary <-> tensor, execution provider and optimisation mapping + src/constants.rs atoms and execution provider names + .cargo/config.toml rpath and macOS link flags (shipped in the Hex package) +config/config.exs Dev-only: per-OS cargo features. Ignored when Ortex is a dependency. +test/ ExUnit tests (dtype, shape ops, serving); resnet50 tests are opt-in +models/tinymodel.onnx Small 2-input/3-output test model (regenerate: python/multi_input.py) +python/ Model export scripts (PyTorch); write into the current directory +examples/ distilbert and stablelm scripts (need dev deps and exported models) +docs/ Architecture and maintenance docs for agents +TODO.md Review findings, GitHub issue/PR triage, roadmap +``` + +## Setup + +* `mix.exs` declares `elixir: "~> 1.14"` (no OTP constraint). CI runs Elixir 1.17.3 / OTP 27.1.2; + Elixir 1.20.4 / OTP 29.1.2 is verified to work. +* A stable Rust toolchain (verified with 1.99.1). There is no precompiled NIF yet. +* Network access on the first build: `ort`'s build script downloads ONNX Runtime from + `parcel.pyke.io`. Offline or unusual targets need `ORT_LIB_LOCATION` (see + [`docs/build-and-packaging.md`](docs/build-and-packaging.md)). +* Python with `torch`/`torchvision`/`onnx` only if you regenerate test models. + +## Commands + +```sh +mix deps.get +MIX_ENV=test mix compile --warnings-as-errors # builds the NIF too +mix test # resnet50 tests auto-excluded without models/resnet50.onnx +mix format --check-formatted +(cd native/ortex && cargo fmt --check && cargo clippy --all-targets) +``` + +Use `MIX_ENV=test` for library work. The `:dev` environment pulls `exla`, `torchx` and +`tokenizers`, which download large native artifacts; `exla` and `torchx` are not used anywhere. +That also makes `mix docs` slow until the dev deps are trimmed (see `TODO.md`). +`mix test --warnings-as-errors` currently fails after a passing run because `test/test_helper.exs` +uses `IO.warn` when `models/resnet50.onnx` is missing (TODO TEST-08). + +## Gotchas (read before changing code) + +1. **Rust changes are not picked up by `mix compile`.** `lib/ortex/native.ex` builds the crate + itself through Rustler's private `compile_crate` and then sets `skip_compilation?: true`, so Mix + does not track the Rust sources. Run `mix compile --force` (or touch `lib/ortex/native.ex`) after + editing anything under `native/`. With Rustler 0.37+ (what new installs resolve) the NIF is + written to `priv/native/ortex.so` in the source tree and cargo builds in `native/ortex/target/`; + do not commit either. +2. **A NIF is declared in three places** that must agree: the `#[rustler::nif]` function in + `native/ortex/src/lib.rs`, the list inside `rustler::init!` (Rustler 0.29 needs it), and the + stub in `lib/ortex/native.ex` (same name and arity, body `:erlang.nif_error(:nif_not_loaded)`). +3. **NIF errors are values.** Rust returns `rustler::Error::Term(Box::new(msg))`, which arrives in + Elixir as `{:error, msg}`. Every Elixir caller must match it. Decoding failures arrive as + `ArgumentError`; Rust panics arrive as `ErlangError :nif_panicked`. +4. **No panics on user-reachable paths.** No `unwrap`, `expect`, `panic!`, `todo!` or + `unimplemented!` on data that came from Elixir or from a model. Return an error instead. +5. **Tensors are immutable and contiguous.** `OrtexTensor` arrays must be standard (row-major) + layout and must never be mutated: `to_binary` hands out zero-copy BEAM binaries that alias them. +6. **Schedulers.** Anything that can take more than about 1 ms (copies of large arrays, file IO, + inference) must be a dirty NIF (`schedule = "DirtyCpu"` or `"DirtyIo"`). +7. **Bool.** Nx has no boolean type. ONNX bool outputs are returned as `{:u, 8}`; `{:u, 8}` inputs are + converted to bool only when the model input is bool. +8. **Dependency config is not loaded.** `config/config.exs` only affects development of this repo. + Applications that depend on Ortex set `config :ortex, Ortex.Native, features: [...]` themselves. +9. **`ort` and `ndarray` move together.** `ort`'s ndarray conversions only accept the `ndarray` + major version `ort` itself depends on. Bump both at once. +10. **`ort` is pre-2.0 with breaking changes between release candidates.** Pin an exact version + (`=2.0.0-rc.N`; a caret requirement lets `cargo update` jump RCs) and read the notes of every + intermediate RC. Rules that bite during the rc.15 upgrade (details in `docs/upgrading.md` 3.3): + only one ONNX Runtime `Environment` per OS process, ever; `Session::run` needs `&mut self`; + `GraphOptimizationLevel::Level3` no longer means "all" (use `All`); CPU-only sessions need + `AutoDevicePolicy::PreferCPU`; `half` must be enabled as a feature. +11. **Execution providers fail silently.** An unknown atom, or a provider that is not compiled in + or not installed, quietly runs on CPU. Do not rely on `Ortex.load(path, [:cuda])` raising. + +## Design rules + +All code follows Responsibility-Driven Design: every module owns its data and behaviour, and +collaborators ask it to act instead of reading or duplicating its state. Before writing code, state +each new or changed module's responsibilities (what it knows, what it does) and its +collaborators, and check that nothing is split or misplaced. The current ownership map and the +known violations are in [`docs/architecture.md`](docs/architecture.md) sections 3 and 9. In practice: + +* `Ortex.Backend` owns the shape of its tensor data. Other modules must ask it to wrap or unwrap + NIF references instead of pattern-matching `%Ortex.Backend{}` inside `%Nx.Tensor{}`. +* The native layer owns everything about how ONNX Runtime is built, linked and which execution + providers exist. Elixir asks the NIF; it does not mirror Cargo features or build directories. +* A tensor knows its own dtype and shape. Do not pass them from Elixir when Rust can read them. +* The NIF boundary owns error translation: one place converts native results into Elixir errors. + +## Recipes + +**Add a NIF.** Write the Rust function in `lib.rs` (choose a scheduler), add it to `rustler::init!` +if the Rustler version needs the list, add the Elixir stub to `lib/ortex/native.ex`, call it from +the owning Elixir module, handle `{:error, msg}`, add a test, run `mix compile --force`. + +**Add a dtype.** Add a variant to `OrtexTensor` and to every `match` in `tensor.rs` (`shape`, `reshape`, +`dtype`, `to_bytes`, `slice`, `Clone`, both `TryFrom` impls, `concatenate`), extend +`utils::from_binary`, add the atom to `constants.rs` if needed, extend `test/dtype/dtype_test.exs` +and the concat test table. Check `docs/elixir-layer.md` section 3 for the Nx/ONNX mapping. + +**Add or change an execution provider.** Cargo feature in `native/ortex/Cargo.toml`, name in +`constants.rs`, mapping in `utils::map_eps`, docs in `docs/execution-providers.md`. Remember that +`ort` rc.13+ only compiles a provider's struct when its Cargo feature is enabled. + +**Add a session option.** Prefer a keyword option on `Ortex.load/2` (see the unified design in +`TODO.md`) mapped to one `SessionBuilder` call in `model.rs`. Do not add more positional arguments. + +**Upgrade `ort` / ONNX Runtime or Elixir dependencies.** Follow [`docs/upgrading.md`](docs/upgrading.md). + +## Tests and verification + +* Every change: `mix format`, `MIX_ENV=test mix compile --warnings-as-errors`, `mix test`. + Rust changes also: `cargo fmt` and `cargo clippy` in `native/ortex`. +* Bug fixes come with a regression test. Many current bugs are listed with reproduction steps in + [`docs/testing-and-ci.md`](docs/testing-and-ci.md#4-behaviour-probe-results). +* Keep committed test models tiny and commit the script that generates them. + +## Pull requests + +* Keep PRs small and focused; split dependency bumps, refactors and features. When work depends on + an unmerged layer, use stacked PRs where each layer builds and passes tests on its own. +* Update `docs/` when behaviour or architecture changes, and tick or edit items in `TODO.md`. +* Never commit `_build/`, `deps/`, `native/ortex/target/` or large `.onnx` files. diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..47ec280 --- /dev/null +++ b/TODO.md @@ -0,0 +1,546 @@ +# Ortex TODO: review findings and roadmap + +Snapshot: 2026-10-29. Baseline: `main` at `55dc3de` (Hex `0.1.10` + PR #48). Architecture background +for every item is in [`docs/`](docs/README.md); contributor rules are in [`AGENTS.md`](AGENTS.md); the +step-by-step upgrade guide is [`docs/upgrading.md`](docs/upgrading.md). + +**How these findings were produced.** A full read of every Elixir, Rust, test, config, CI and example file. +`main` was built and tested on macOS arm64 with Elixir 1.20.4 / OTP 29.1.2 / Rust 1.99.1. A fresh +consumer install was simulated (a new Mix app depending on Ortex, with fresh dependency resolution). +Dependency-bump probes covered Nx 1.0.0, Hex Rustler 0.38 and crate Rustler 0.39. Edge cases and +memory behaviour were probed with scripts against the baseline build. All open pull requests were +reviewed statically; code from forks was not executed. All 25 GitHub issues were read with their +comments. Upstream release notes and sources were checked for `ort`, ONNX Runtime, ONNX, Nx, +Rustler and rustler_precompiled. Line numbers refer to the baseline commit. + +**Legend.** Priority: **P0** hurts users today or can crash or corrupt; **P1** needed for the upgrade, or a +major gap; **P2** important improvement; **P3** cleanup. IDs are stable. Tick the box when an item is +done and leave it in place with a note. "Verified" means the behaviour was reproduced by running code. + +--- + +## 1. Answers in brief + +### 1.1 What is left to get onto the latest Elixir + +The code itself already runs on the newest Elixir. On Elixir 1.20.4 / OTP 29.1.2, `main` compiles with +`--warnings-as-errors` and passes all 117 tests. A fresh application can install it and run a model, +resolving Nx 0.13.1 and Rustler 0.38.0. Bumping to `nx ~> 1.0`, `rustler ~> 0.38` and `ex_doc ~> 0.40` +also passes all tests with no Elixir code changes (verified). What is left: + +1. **Release.** Hex `0.1.10` predates PR #48, so `inspect(model)` is broken for Hex users on Elixir 1.19+ + (BUG-13). It also requires `{:nx, "~> 0.6"}`, which cannot resolve next to Nx/EXLA/Bumblebee 1.0 + (COMPAT-01). +2. **Nx contract gaps that newer Nx exposes.** `Nx.stack/2` raises on Ortex tensors since Nx 0.8 + (BUG-14). Mixed-backend `Nx.concatenate/2` raises `MatchError` (BUG-15). Partitioned serving crashes + (BUG-08). There are also the older gaps: `backend_copy`, `backend_deallocate`, `to_binary` limits, + and dynamic slices. +3. **Rustler.** + - The crate is at 0.29. Moving to 0.39 means three layers of mechanical compile errors (COMPAT-02). + - Hex Rustler 0.37+ moved the cargo build directory, which silently broke `copy_ort_libs` for + dynamically linked (GPU) builds (BUILD-01). + - `lib/ortex/native.ex` should use plain `use Rustler` instead of the private + `Rustler.Compiler.compile_crate` (COMPAT-07). +4. **Floors, pins and CI.** Raise the Elixir floor to 1.17 (Nx 1.0's floor). Unpin `ex_doc 0.29.4`. CI + currently covers only Elixir 1.17 / OTP 27 (COMPAT-03, COMPAT-04, CI-02). + +### 1.2 What it takes to reach the latest ONNX Runtime + +The move is from `ort 2.0.0-rc.8` (ONNX Runtime 1.19.2, max opset 21) to `ort 2.0.0-rc.15` (ONNX +Runtime 1.31.0, opset 26). ONNX Runtime 1.31.1 (2026-10-26) is not bundled by any `ort` release yet. +`ort` has not reached 2.0.0; waiting for it has stalled Ortex since 2025 (issue #45), so pin an exact RC. + +Seven release candidates of breaking changes touch every Rust file: + +- module paths; +- `Session::run(&mut self)`, which forces a concurrency decision; +- a mandatory `Environment` that `ort` asks libraries to let users supply; +- execution providers gated at compile time, with strict matching against the downloadable binaries; +- `GraphOptimizationLevel::Level3` no longer meaning "all"; +- new tensor element types, `ndarray 0.17`, CUDA 13 only, no Intel macOS binaries; +- a minimum Rust of 1.92. + +PR #47 already does most of the work up to rc.11. + +Measured effort: from `main`, a compile probe needed about 172 changed lines to reach a clean +`cargo check` against rc.15. From PR #47 the remaining delta is about four import fixes, provider +`#[cfg]` gating, and the environment lifecycle (ONNX Runtime allows exactly one environment per OS +process). The upgrade also raises the platform floor of the prebuilt binaries: + +- glibc 2.39 / libstdc++ 13.2 on Linux; +- AVX2 on x86_64; +- arm64-only macOS 13.4+; +- CUDA 13.2+ with cuDNN 9.23+. + +Older Linux hosts would need Microsoft's ONNX Runtime binaries (glibc 2.28+) linked through +`ORT_LIB_PATH` or loaded with `load-dynamic`. + +One item is time-sensitive. Rust 1.100, the next stable (about 2026-11-12), changes cargo's build-dir +layout. `ort` warns that versions before rc.14 "will fail to build when `copy-dylibs` is enabled". +Reproduced on Rust 1.100 beta: dynamically linked builds of `main` fail, and that includes this repo's +own Linux CI configuration (BUILD-10). The full checklist is in section 3.2 and `docs/upgrading.md`. + +### 1.3 What is missing for complete ONNX support + +- Tensor types beyond the 12 numeric ones: string, complex, and the 8-, 4- and 2-bit families. Today + they panic. +- Non-tensor values (sequences, maps, optionals), which also panic. +- Named inputs and output selection. +- Session options (three competing PRs: #50, #51, #52). +- Loading from memory (issue #29). The API already exists in `ort`. +- Model metadata. +- Custom-op libraries and onnxruntime-extensions. Issue #32's contrib ops are already present in ONNX + Runtime 1.19.2. +- Execution-provider options and availability reporting. +- Device-resident outputs (IoBinding), run cancellation, profiling, logging and telemetry. +- Models above opset 21, which needs the upgrade. + +Section 3.5 has the full list. + +--- + +## 2. Roadmap + +Ordered so each step ships on its own. Use stacked PRs where a step depends on an unmerged one. + +| Phase | Goal | Items | Existing PRs | +| --- | --- | --- | --- | +| 0 | **Patch release 0.1.11 on the current `ort`**: unblock Nx 1.0, Elixir 1.19+ inspect, crashes, GPU loading | BUG-13, COMPAT-01, BUG-14, BUILD-01, SAFE-01, SAFE-02, SAFE-05, BUILD-10, BUG-02, BUG-04, BUG-05, BUG-07, BUG-08, TEST-01, CI-01, CI-02, DOC-01 | pieces of #47 | +| 1 | **Modernise without changing `ort`** (the first half of #47, which the maintainer asked to split) | COMPAT-02 to COMPAT-07, BUG-01, BUG-03, BUG-06, BUG-15, BUG-16, BUILD-05 to BUILD-09, BUILD-11, REPO-* | #47 (part A) | +| 2 | **`ort` rc.15 / ONNX Runtime 1.31** | ORT-01 to ORT-17, SAFE-06, PERF-01, PERF-04 | #47 (part B, retargeted to rc.15); close #46 | +| 3 | **Session options API** | ARCH-07, PERF-05 | #52 as base, absorbing #50 and #51 | +| 4 | **Precompiled NIFs and platforms** | BUILD-02 to BUILD-04, CI-05 | #49, rebased last (it can ship earlier on rc.8 if phase 2 stalls) | +| 5 | **Complete ONNX support** | ONNX-01 to ONNX-16, ARCH-08 | | +| 6 | **Architecture and performance follow-ups** | ARCH-01 to ARCH-11, PERF-02, PERF-03, PERF-06 to PERF-08 | | + +--- + +## 3. Findings + +### 3.1 Elixir ecosystem compatibility + +- [ ] **COMPAT-01 (P0) Allow Nx 1.0.** + - `mix.exs:35` has `{:nx, "~> 0.6"}`, i.e. `< 1.0.0`. Nx, EXLA and Torchx 1.0.0 shipped 2026-09-10. + - Verified: the suite passes on Nx 1.0.0 unchanged, because `Ortex.Backend` generates stubs from + `behaviour_info`. That covers callbacks added in 1.0 (`stack/3`, `to_pointer/2`, `from_pointer/5`, + `block/4`), callbacks removed (21), and arity changes (`gather/4`, `indexed_add/5`, `indexed_put/5`). + - For 0.1.11 use `"~> 0.6 or ~> 1.0"` and test both ends. Narrow it later. + - PR #47's `"~> 0.10"` still excludes 1.0. +- [ ] **COMPAT-02 (P1) Upgrade the Rustler crate 0.29 to 0.39.** These are the probe's three error layers, in order: + 1. Rustler 0.38 removed the NIF list in `rustler::init!` (`lib.rs:107-118`), and the `rustler::resource` module is private, so import `rustler::ResourceArc` from the root. + 2. Rustler 0.38 removed `rustler::resource!` (`lib.rs:119-123`). Use `#[rustler::resource_impl] impl rustler::Resource for OrtexModel {}` and the same for `OrtexTensor`. + 3. `ort::Session` is not `RefUnwindSafe`; the bound has existed since Rustler 0.35. Add `impl std::panic::RefUnwindSafe for OrtexModel {}`. + + Crate MSRV is 1.91. NIF version defaults to 2.15, which loads on OTP 22 to 29. +- [ ] **COMPAT-03 (P1) Raise the Elixir floor.** `mix.exs:8`: `elixir: "~> 1.17"` (Nx 1.0's floor). OTP 29 needs Elixir 1.20. +- [ ] **COMPAT-04 (P2) Unpin ex_doc.** `mix.exs:37` exact `0.29.4` to `~> 0.40` (`mix docs` verified clean with 0.40.4). +- [ ] **COMPAT-05 (P2) Refresh the lock.** Nx 0.6.2 emits about 890 deprecation warnings on Elixir 1.20, + including unpinned bitstring `size(...)`, which is a future hard error. Remove stale `mix.lock` entries. +- [ ] **COMPAT-06 (P3) Use built-in `JSON`** (Elixir 1.18+) in `examples/distilbert` instead of the undeclared `Jason`. +- [ ] **COMPAT-07 (P1) Stop calling Rustler's private API.** + - `lib/ortex/native.ex:4-24` calls `@doc false` `Rustler.Compiler.compile_crate` behind a version switch and then sets `use Rustler, skip_compilation?: true`. That also means Mix never sees the Rust sources (BUILD-05). + - Replace it with `use Rustler, otp_app: :ortex, crate: :ortex` followed by the library copy step. + - Verified: tests pass and Rust edits then trigger a rebuild. + - Declare `{:rustler, "~> 0.38", runtime: false}`. + +### 3.2 ONNX Runtime upgrade (`ort` rc.8 to rc.15) + +Code-level detail is in `docs/upgrading.md`. Do these as one PR stack after phase 1, starting from PR #47's +`ort` commit (`9a9b8f2b`) and crediting #46. + +- [ ] **ORT-01 (P1) Pin `ort = { version = "=2.0.0-rc.15", features = ["half"] }`.** + - `half` has not been a default feature since rc.10. + - Bump `ndarray = "0.17"` in the same change. + - Today's caret requirement (`"2.0.0-rc.8"`) allows later RCs, so a plain `cargo update` would jump to rc.15 and break the build. +- [ ] **ORT-02 (P1) Module paths (rc.9, rc.11, rc.12).** Only `ort::{Error, ErrorCode, Result, init, init_from}` remain at the crate root. Imports now come from: + - `ort::session::{Session, SessionInputValue, SessionOutputs, RunOptions}` + - `ort::session::builder::{GraphOptimizationLevel, AutoDevicePolicy}` + - `ort::value::{Value, DynValue, ValueType, TensorElementType, Tensor, TensorRef, Outlet}` (`ort::tensor` merged into `ort::value` in rc.12, which breaks PR #47's import) + - `ort::ep::{ExecutionProviderDispatch, CPU, CUDA, ...}`, renamed from `execution_providers::XXXExecutionProvider`. +- [ ] **ORT-03 (P1) `Session::run` takes `&mut self` (rc.10+).** + - Pick a concurrency model. + - PR #47's `Mutex` serialises every inference on a model, a regression from today's parallel runs (verified parallel on rc.8). + - Options: a small session pool per model, or one session per `Nx.Serving` partition. + - Remove `unsafe impl Sync` (`model.rs:30`). +- [ ] **ORT-04 (P1) Mandatory `Environment` (rc.14), one per OS process.** + - Sessions are built with `Session::builder(&env)`. + - A second `ort::init().build()` fails even after the first environment is dropped (verified), so per-model environments are impossible. + - Build it once in the NIF (in the `load` callback with options from app config, or lazily) and keep it in a `static OnceLock`. A static environment with leaked sessions exited cleanly in 3 of 3 runs on macOS. + - Expose its options through application config: global thread pool, logger bridge to `Logger`, `with_telemetry(false)`. + - Design: `docs/upgrading.md` section 3.3. +- [ ] **ORT-05 (P1) Execution providers are gated at compile time (rc.13).** + - Wrap each `ort::ep::*` arm in `#[cfg(feature = ...)]`, and return a clear error for providers that were not compiled in. + - `ort`'s "unsatisfiable features" check is a deliberate unresolved symbol, which `-undefined dynamic_lookup` hides on macOS. A bad combination builds a NIF without ONNX Runtime that only fails at load (verified). So: drop the flag (BUILD-07), validate feature/target combinations in `native.ex`, and load the NIF in CI per target. + - Since rc.12 the default `AutoDevicePolicy::MaxEfficiency` can put a `[:cpu]` session on CoreML with ONNX Runtime 1.31 (pykeio/ort#662). Use `PreferCPU` when only `:cpu` is requested. + - Drop `:rocm` (ONNX Runtime 1.23 removed it; use MIGraphX, prebuilt on x64 Linux since rc.14). ACL is deprecated in 1.31. +- [ ] **ORT-06 (P1) The optimisation level was renumbered.** + - In rc.8, `Level3` is `ORT_ENABLE_ALL`. In rc.11 and later, `Level3` is `ORT_ENABLE_LAYOUT`, and a new `All` is `ORT_ENABLE_ALL`. Both verified in source. + - Map Ortex's default `3` to `All`, or the default silently loses optimisations. PR #47 maps it to `Level3`. +- [ ] **ORT-07 (P1) Introspection API.** + - Inputs and outputs are accessed through methods, with `name()` and `dtype()`. + - `ValueType::Tensor` carries `shape` and `dimension_symbols`. + - Snapshot this metadata at load time so `show_session` never touches the session (ORT-12). +- [ ] **ORT-08 (P1) Extraction API.** `try_extract_tensor` became `try_extract_array`. Match new element types with a fallback arm instead of exhaustively. +- [ ] **ORT-09 (P2) Zero-copy inputs.** `TensorRef::from_array_view` (rc.10+) replaces cloning every input. It needs standard layout, which `OrtexTensor` already guarantees. +- [ ] **ORT-10 (P1) Platform floor of the prebuilt binaries.** + - Linux: glibc >= 2.39 and libstdc++ >= 13.2, i.e. Ubuntu 24.04+ and Debian 13+ (per `ort` docs and symbol inspection; rc.8 needed about glibc 2.32). + - x86_64 needs AVX2. + - macOS: arm64 only, 13.4+. No Intel macOS since rc.11, which ends issue #34's platform. + - No musl, armv7 or riscv. + - CUDA 13.2+ and cuDNN 9.23+ (no CUDA 12 since rc.13): Turing or newer, so Pascal and Volta are out. + - rc.14 adds Blackwell x64 and aarch64 Jetson/Grace CUDA builds (helps #45 and #41). + - For older Linux, offer Microsoft's ONNX Runtime 1.31.1 (glibc 2.28+) via `ORT_LIB_PATH` + dynamic link or `load-dynamic`. Re-check the target list before promising platforms (BUILD-02). +- [ ] **ORT-11 (P1) Build requirements.** + - rc.14+ needs Rust 1.92. + - rc.10+ downloads through native TLS, so Linux builds need OpenSSL headers. The `tls-rustls` feature avoids that. + - `ORT_LIB_LOCATION` was renamed `ORT_LIB_PATH` in rc.12. rc.15 still reads the old name, but it will not survive into 2.0.0. + - Update the Docker guidance (issue #43). +- [ ] **ORT-12 (P2) `show_session` must not lock.** It is a normal-scheduler NIF; behind a mutex it would block a scheduler during inference. PR #47 has this bug. +- [ ] **ORT-13 (P2) `api-*` features (rc.12).** Defaults use the newest API level. With `default-features = false`, enable the matching `api-NN`. +- [ ] **ORT-14 (P2) Remove the `rustls` pin** (`Cargo.toml:20`). Nothing uses it, and it forces `ring` to build for the target, which breaks Nerves (issue #41). Also remove `tracing-subscriber` and `num-traits`. +- [ ] **ORT-15 (P2) Re-validate packaging.** + - Every pyke package is now a static `libonnxruntime.a`. Verified: the CPU NIF is self-contained, 27.4 MB on macOS arm64. + - CUDA packages add provider libraries loaded at run time: `libonnxruntime_providers_{shared,cuda,tensorrt,nv_tensorrt_rtx}.so`, with the CUDA one at 121 MB. They must sit next to the NIF, and copying them is the only job left for `copy_ort_libs/0`. + - Re-check the precompiled matrix (PR #49). +- [ ] **ORT-16 (P3) Bring-your-own ONNX Runtime.** Offer `load-dynamic` (`ort::init_from(path)` or an absolute `ORT_DYLIB_PATH`) so users can run a newer ONNX Runtime (e.g. 1.31.1), or Microsoft's builds for older glibc, without waiting for `ort`. +- [ ] **ORT-17 (P2) Thread spinning is off by default in pyke builds since rc.11** (client-package build). rc.8 spins by default. For servers, expose `with_intra_op_spinning`/`with_inter_op_spinning` or the global pool's `with_spin_control`, and benchmark before and after the upgrade. + +### 3.3 Correctness bugs + +All "verified" items were reproduced on the baseline build (`docs/testing-and-ci.md` section 4). + +- [ ] **BUG-01 (P1) Model inspect output does not match the docs (verified).** + - `model.rs:54-79` returns `format!("{:#?}", value_type)`, a multi-line `Tensor { ty: Float32, dimensions: [...] }`, and `-1` for dynamic axes. + - The README and `Ortex.Model` docs promise `{"x", "Int32", [nil, 100]}`. + - Return structured data and format it in Elixir (ARCH-05). +- [ ] **BUG-02 (P1) Input count is not validated (verified).** + - `model.rs:92` zips inputs with the model inputs, so extra inputs are silently dropped. + - Missing inputs fail inside a kernel with an unrelated message. + - Validate the count and name the expected inputs. PR #47 adds the count check. +- [ ] **BUG-03 (P1) Execution providers fall back silently (verified; issue #17).** + - Unknown atoms map to CPU (`utils.rs:105`). + - Registration failures are only `tracing::warn!`ed by `ort`, and Ortex installs no subscriber, so `[:cuda]` without CUDA quietly runs on CPU. + - Strings raise a bare `ArgumentError`. + - Fix: reject unknown atoms (PR #47), surface registration failures (`ExecutionProviderDispatch::error_on_failure()` for providers the user asked for, or a Logger warning), and add `Ortex.available_execution_providers/0`. +- [ ] **BUG-04 (P1) `Nx.to_binary/2` ignores `limit` (verified).** + - `backend.ex:44-49`, `utils.rs:84-91`. Nx expects exactly the first `limit` elements. + - Fix in Rust from the tensor's own element size, or zero-copy in Elixir with `binary_part/3` on the resource binary. +- [ ] **BUG-05 (P1) `backend_copy/3` and `backend_deallocate/1` raise (verified).** + - Nx.Serving's distributed docs recommend `Nx.backend_copy(&1, Nx.BinaryBackend)`, which fails on Ortex outputs. + - Make `backend_copy` delegate to the transfer logic, and have `backend_deallocate` return `:ok`. +- [ ] **BUG-06 (P2) `Nx.slice/4` with tensor start indices raises `ArgumentError` (verified; issue #8).** + - Nx turns every index into a scalar tensor when any is dynamic and expects the backend to clamp to `[0, dim - len]` (`backend.ex:81-85`). +- [ ] **BUG-07 (P1) NIF errors are ignored in shape ops.** + - `slice`, `reshape`, `squeeze` and `concatenate` (`backend.ex:81-125`) store a `{:error, msg}` tuple as the tensor ref. + - Match and raise (PR #47). +- [ ] **BUG-08 (P1) `Ortex.Serving.init/3` accepts exactly one partition (verified).** + - `serving.ex:62` matches `[_defn_options]`. `partitions: true` gives one option list per partition (`max_concurrency` with the Evaluator, one per device with EXLA), so it raises `FunctionClauseError`. + - Accept any list (PR #47). +- [ ] **BUG-09 (P3) Optional Nx callbacks are stubbed to raise** (`backend.ex:141`). This blocks Nx fallbacks on Nx < 1.0. Nx 1.0 has no optional callbacks, so the item disappears once the floor is 1.0. +- [ ] **BUG-10 (P2) Shape NIFs copy whole arrays on normal schedulers** (`lib.rs:73-105`: `slice`, `reshape`, `concatenate`; issues #8, #11). Mark them `DirtyCpu`, or make reshape and squeeze metadata-only. +- [ ] **BUG-11 (P2) Bool inputs panic on common tensors.** + - A model `bool` input given `Nx.tensor([1, 0])` (`s32` on Nx 0.9+) or a `u8` value other than 0 or 1 panics in `tensor.rs:187-203`. + - Accept any integer type (non-zero means true), or return an error that names the input. +- [ ] **BUG-12 (P3) Small dead or fragile code.** + - `backend_transfer(tensor, Nx.Tensor, _)` is a dead clause (`backend.ex:51-54`). + - `@derive {Nx.Container, containers: [:ref]}` is unused, and wrong for a reference (`backend.ex:16`). + - `inspect/2` relies on `min(:infinity, n)` term ordering (`backend.ex:73-76`). + - The `:dnnl` atom is declared but unmapped (`constants.rs:20`). + - Levels outside 1..3 silently disable optimisation (`utils.rs:111-118`). + - `squeeze/3` recomputes a shape Nx already put in `out` via the private `Nx.Shape.squeeze/3`. +- [ ] **BUG-13 (P0) Hex 0.1.10 has a broken `inspect(model)` on Elixir 1.19+.** Elixir 1.19 changed `Inspect.List.inspect/2`'s return value. PR #48 fixed it on `main`, but there has been no release since. Release 0.1.11. +- [ ] **BUG-14 (P1) `Nx.stack/2` raises on Ortex tensors (verified with Nx 1.0; affects Nx 0.8+).** + - Since Nx 0.8, `stack` is a backend callback instead of `new_axis` plus `concatenate`. + - Fresh installs already resolve Nx 0.13.1, so this affects users today. + - Implement `stack/3` with reshape plus the concatenate NIF. +- [ ] **BUG-15 (P2) Mixed-backend `Nx.concatenate/2` raises `MatchError` (verified with Nx 1.0).** + - Nx 1.0 hands mixed Binary/Ortex lists to the non-Binary backend (`backend.ex:116-120`); mixed types raise `RuntimeError`. + - Transfer and cast inputs instead. +- [ ] **BUG-16 (P1) `Ortex.Serving` runs inference inside `handle_batch/3`** (`serving.ex:68-72`). Nx.Serving calls it in its GenServer and runs the returned function in a supervised task. Ortex's blocking call stalls the serving process, and a raise kills the whole serving instead of one batch. Move materialisation and `Ortex.run/2` into the `:execute` function. + +### 3.4 Memory safety and crashes + +- [ ] **SAFE-01 (P0) Unchecked raw-pointer view over BEAM binaries.** + - `utils.rs:16-20, 41-78` builds `ArrayViewMut::from_shape_ptr(shape, bin.as_ptr() as *mut T)`. + - There is no length check (a short binary reads out of bounds), no alignment guarantee for sub-binaries, and it creates a mutable view of immutable memory. + - Validate `len == product(shape) * size_of::()` and copy into a `Vec` (PR #47's `array_from_binary`). +- [ ] **SAFE-02 (P0) Panics reachable from Elixir (verified for `c64`, `s4`, `u2`, `f8`, `f8_e4m3fn`).** Each becomes `ErlangError :nif_panicked`, with the message only on stderr. Return errors (PR #47 covers most) and raise `ArgumentError` in Elixir for unsupported types. The sites: + - `utils.rs:78` `unimplemented!()` for unsupported Nx types; + - `tensor.rs:239` `panic!` for sequence and map outputs; + - `tensor.rs:279-281` `todo!()` for string outputs; + - `tensor.rs:187-203` bool conversion. +- [ ] **SAFE-03 (P2) Remove `unsafe impl Sync for OrtexModel`** (`model.rs:30`). It is redundant on rc.8 (`ort` already makes `Session` `Send + Sync`) and unsound once `run` needs `&mut self`. +- [ ] **SAFE-04 (P3) Clean up `get_bytes`** (`tensor.rs:219-228`). It uses `std::mem::zeroed()` to learn an element size and assumes standard layout. Use `size_of::()` with `as_slice_memory_order()` or `bytemuck`. Add `// SAFETY:` comments to every `unsafe` block. +- [ ] **SAFE-05 (P0) Loading a large model can crash the VM (issue #42).** + - ONNX Runtime graph optimisation at level 3 overflowed the C stack while loading a 300 MB model: a silent BEAM exit on Windows, a segfault on WSL. + - `init` runs on a dirty IO scheduler, whose threads default to 40 kilowords of stack (320 KiB on 64-bit; `+sssdio`). + - Build the session on a dedicated Rust thread with a large, configurable stack (`std::thread::Builder::stack_size`) and join it. + - Document `+sssdio` and the optimisation-level workaround meanwhile. +- [ ] **SAFE-06 (P1) Concurrent `run` on one session is unsafe for some providers.** + - Today `Ortex.run/2` calls `Session::run(&self)` on one shared session from many dirty schedulers. That is legal in rc.8, and verified to work on the CPU provider. + - `ort` changed `run` to `&mut self` because "previous versions of `ort` that allowed concurrent inference often saw crashes and memory corruption". + - Upstream, ONNX Runtime forbids concurrent `Run()` on one session with DirectML and CUDA graphs. + - Until the upgrade (ORT-03), serialise runs per model when a non-CPU provider is active, or document the restriction. + +### 3.5 Complete ONNX support + +| ID | Pri | Gap | Notes | +| --- | --- | --- | --- | +| ONNX-01 | P1 | Models above opset 21 / IR 10 cannot load | Needs the ONNX Runtime upgrade (opset 26 on 1.28 to 1.31). Issues #16 and #36 were early versions of this. | +| ONNX-02 | P2 | String tensors (in and out) | Common in NLP and scikit-learn models. Represent as lists of binaries. Today output hits `todo!()`. rc.15: `try_extract_strings`, `try_extract_string_array`, `Tensor::from_string_array`. | +| ONNX-03 | P2 | Non-tensor values: sequences, maps, optionals | scikit-learn classifiers (ZipMap) return `seq(map(int64, float))`, which panics today. Return Elixir lists and maps. rc.15: `try_extract_sequence`, `try_extract_map`/`try_extract_key_values`; rc.13 fixed a panic when committing sessions with map inputs/outputs. | +| ONNX-04 | P2 | `FLOAT8E5M2` and `FLOAT8E4M3FN` (Nx `{:f, 8}`, `{:f8_e4m3fn, 8}`); `INT4`/`UINT4`, `INT2`/`UINT2` (Nx `{:s, 4}`, `{:u, 4}`, `{:s, 2}`, `{:u, 2}`) | Nx 1.0 has all of these. Mind sub-byte packing order (ONNX packs two 4-bit values per byte, low nibble first) and that Nx accepts sub-byte bitstrings since 1.0. `ort` has no Rust element types for these: build `DynTensor`s from raw bytes. | +| ONNX-05 | P3 | `COMPLEX64`/`COMPLEX128`, `FLOAT8E4M3FNUZ`, `FLOAT8E5M2FNUZ`, `FLOAT4E2M1`, `FLOAT8E8M0` | Complex maps to Nx `{:c, 64}`/`{:c, 128}` through `ort`'s `num-complex` feature (ONNX Runtime's complex kernel coverage is small). The others have no Nx type: return a clear "unsupported" error. | +| ONNX-06 | P2 | Named inputs and output selection | Accept `%{"input_ids" => t}`. Add an option to request specific outputs (ONNX Runtime then skips work), and optionally return a map. | +| ONNX-07 | P1 | Session options | Threads, execution mode, spinning, memory pattern, arena, deterministic compute, config entries, optimised-model path, free-dimension overrides. See ARCH-07. | +| ONNX-08 | P2 | Load from memory (issue #29) | `SessionBuilder::commit_from_memory(&[u8])` exists in rc.8 and rc.15 (rc.15 also has a borrowing `commit_from_memory_directly` and a `canceler()` for aborting slow loads); about 20 lines plus a dirty NIF. The maintainer's "wait for ort 2.0" reason no longer applies. | +| ONNX-09 | P2 | External data files, ORT-format models | Weights in `*.onnx_data` load from a path only when side by side. From memory they need `with_external_initializer_file`/`..._in_memory`. | +| ONNX-10 | P2 | Model metadata | Producer, version, description, custom metadata map, opset imports, symbolic dimension names. | +| ONNX-11 | P2 | Custom-op libraries, onnxruntime-extensions (issue #32) | `com.microsoft` contrib ops such as `MatMulNBits` and `GroupQueryAttention` are already in the ONNX Runtime 1.19.2 binary (checked on the macOS artifact). Add a CI model using one, then close #32. Custom-op library registration is still missing. | +| ONNX-12 | P2 | Execution-provider options and availability | Device id, memory limits, TensorRT engine cache, CoreML flags. Prior art: the unmerged branch `tm-execution-providers`. | +| ONNX-13 | P3 | IoBinding / device-resident outputs | Keep data on the GPU between calls (`create_binding` + `run_binding`; open bug pykeio/ort#667: `bind_input` can deadlock in rc.15 for borrowed tensors already on the device). | +| ONNX-14 | P3 | Run options | Cancel long inferences (`RunOptions::terminate`), run tags, per-run log level, output selection (`with_outputs`) through `run_with_options`. | +| ONNX-15 | P3 | Profiling and logging | Route ONNX Runtime logs to Logger (`model.rs:39` TODO); expose profiling output. | +| ONNX-16 | P3 | Telemetry | `:telemetry` spans for load and run, like the rest of the Nx ecosystem. | + +### 3.6 Build, packaging, distribution + +- [ ] **BUILD-01 (P0) `copy_ort_libs` finds nothing with current Rustler.** + - Hex Rustler 0.37.0 stopped setting the cargo target directory. Builds now land in `native/ortex/target//`; for a dependency, `deps/ortex/native/ortex/target/release/` (verified with 0.38). + - `util.ex:8-17` only searches `_build//lib/ortex/native/ortex/{release,debug}`, so dynamically linked ONNX Runtime builds (CUDA, TensorRT, a shared `ORT_LIB_LOCATION`) load without their libraries. + - It also misses Windows' `onnxruntime*.dll` (issue #26) and cross-compiled `/` directories (issue #41). + - Short term: ask `cargo metadata` for `target_directory`, then add `/` and the profile. + - Long term: make the native build self-contained (ARCH-02). +- [ ] **BUILD-02 (P1) Precompiled NIFs (issue #27, PR #49).** Section 5 has the PR review. Decide: + - provider coverage per artifact, since CoreML and DirectML need their features; + - GPU variants (`rustler_precompiled` `variants`); + - a glibc baseline (PR #49 builds on ubuntu-22.04, i.e. glibc 2.35+, which excludes Debian 11, Ubuntu 20.04, RHEL 8 and Amazon Linux 2); + - the target list after the upgrade, with no Intel macOS; + - NIF version 2.15 only, which covers OTP 22 to 29; + - the checksum step in the release flow. + + Require `rustler_precompiled >= 0.10.1`: earlier versions fail on GitHub's large response headers. +- [ ] **BUILD-03 (P1) Builds depend on pyke's CDN.** A transient 404 from `parcel.pyke.io` failed PR #50's Linux CI. + - Cache ONNX Runtime downloads in CI. + - Document `ORT_LIB_LOCATION` for offline builds. + - Precompiled NIFs remove the dependency for most users. +- [ ] **BUILD-04 (P1) Cross-compiling and Nerves (issue #41).** + - A working community recipe (`abelino/ortex_fw`) sets `config :ortex, Ortex.Native, target: "aarch64-unknown-linux-gnu", env: [{"CC", ""}, {"CFLAGS", ""}, {"CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER", "aarch64-nerves-linux-gnu-gcc"}]`. + - What Ortex should do: + - derive the triple from Nerves' environment and pass `target:` and `env:` to Rustler, letting user overrides win; + - drop `rustls` (ORT-14); + - fix the copy step for triple directories and the target OS; + - document that 32-bit ARM, musl and riscv64 need a self-built ONNX Runtime via `ORT_LIB_LOCATION`. + - On the device, the static ONNX Runtime still needs `libstdc++`. + - aarch64 Nerves may simply use PR #49's `aarch64-unknown-linux-gnu` artifact; test on hardware. +- [ ] **BUILD-05 (P2) Rust edits do not trigger rebuilds (verified).** `skip_compilation?: true` means no `@external_resource`. COMPAT-07 fixes it. +- [ ] **BUILD-06 (P2) Dev config differs from what users get (issue #17).** + - `config/config.exs:13-18` enables `cuda`/`tensorrt` on Linux, so CI tests the dynamic CUDA build while users get the static CPU build. + - Mix never loads a dependency's config (verified), so shipping `config/config.exs` (`mix.exs:46`) does nothing, and PR #47's `ORTEX_FEATURES` (read in that file) has the same flaw. + - Default to CPU (PR #47 does). + - If an env var is wanted, read it in `lib/ortex/native.ex`. + - Remove the no-op `:dbg_callback` line (`config/config.exs:2-4`). + - Stop shipping the file. +- [ ] **BUILD-07 (P2) Simplify `.cargo/config.toml`.** With the Rustler crate 0.36 or later, macOS needs neither `-undefined dynamic_lookup` nor `-fapple-link-rtlib`; only the rpath flag remains. This was verified with crate 0.39 on macOS arm64. Re-check after the ORT upgrade. +- [ ] **BUILD-08 (P2) Cargo hygiene.** + - `edition = "2018"` to 2021 or 2024. + - Remove the unused `tracing-subscriber`, `num-traits` and `rustls`. + - Set `rust-version`. + - Pin `ort` exactly. +- [ ] **BUILD-09 (P3) Heavy unused dev deps.** `exla` and `torchx` (`mix.exs:38-39`) are unused and make `:dev` builds and `mix docs` download large artifacts. `tokenizers` only serves the examples. Move the examples to `Mix.install` scripts. +- [ ] **BUILD-10 (P0, deadline about 2026-11-12) Rust 1.100 breaks dynamically linked builds (verified).** + - Rust 1.100 ships cargo's new build-dir layout: `OUT_DIR` becomes `target//build///out`, one level deeper. + - `ort-sys` rc.8's `copy-dylibs` locates the target directory as `OUT_DIR.ancestors().nth(3)`, now `target//build`, and `unwrap()`s a symlink into `.../build/examples`, which does not exist. + - Reproduced with Rust 1.100.0-beta.7 and a dynamic ONNX Runtime: `failed to run custom build command for ort-sys v2.0.0-rc.8 ... panicked at build.rs:111:66 ... No such file or directory`. The same build passes on 1.99.1. + - Static CPU builds (what dependents get by default) still build on the beta. + - Affected: + - every `cuda`/`tensorrt` build on x86_64 Linux/Windows, including **this repo's own Linux CI**, because `config/config.exs` enables those features; + - any build with a dynamic `ORT_LIB_LOCATION`. + - Real fix: `ort` rc.14+ (section 3.2). + - Stopgap for 0.1.11: make Linux CI CPU-only, and pin Rust 1.99 for GPU builds (document it, or add `native/ortex/rust-toolchain.toml` for GPU users). +- [ ] **BUILD-11 (P2) Ignore `/priv/native/`.** Rustler 0.37+ writes `priv/native/ortex.so` into the source tree (`_build/.../priv` becomes a symlink), so add `/priv/native/` to `.gitignore`. + +### 3.7 Performance + +- [ ] **PERF-01 (P2) Inputs are cloned into `ort` values on every run** (three copies for bool inputs). Use zero-copy `TensorRef` views after the upgrade (ORT-09). +- [ ] **PERF-02 (P3) Outputs are copied out of ONNX Runtime memory** (`into_owned`). Keeping the `ort` value inside the resource would make the output path zero-copy. +- [ ] **PERF-03 (P3) `from_binary` copies BEAM binaries into Rust.** Revisit only if profiling shows it matters, e.g. by passing borrowed binaries straight into `run`. +- [ ] **PERF-04 (P1, with ORT-03) Do not serialise inference per model after the upgrade.** Use a session pool or per-partition sessions, and benchmark concurrent `Nx.Serving` before and after. +- [ ] **PERF-05 (P2) Thread oversubscription.** Every session has its own intra-op pool with spinning on. Expose thread and spinning options and a process-wide pool (ARCH-07, ORT-04), and document settings for BEAM nodes. Note that `session.intra_op_num_threads` style config keys and `ORT_INTRA_OP_NUM_THREADS` only exist in ONNX Runtime 1.28 and 1.29+. +- [ ] **PERF-06 (P3) Batch stacking uses the default `defn` compiler.** `Ortex.Serving` stacks with whatever compiler is the default (EXLA in many apps, which adds device copies). Use `Nx.Defn.Evaluator` explicitly (PR #47), or `Nx.LazyContainer.traverse/3`. +- [ ] **PERF-07 (P3) `concatenate` copies twice** (`tensor.rs:359-363`). Concatenate into a standard-layout array directly. +- [ ] **PERF-08 (P2) Native tensor memory is invisible to the BEAM GC (issue #44).** + - `ResourceArc::new` accounts about 100 bytes per tensor, while the data is a separate allocation. Dead tensors are freed only when their owner process collects them, and old-heap garbage waits for a full sweep. + - Probe results (macOS, rc.8, 600 runs): + - Fixed shapes: RSS rose about 45 MB, then stayed flat; GC returned it. + - Random shapes: it peaked about 180 MB above baseline, then drifted down. + - No unbounded growth was reproduced. + - Options: + - keep the resource binary in `%Ortex.Backend{}` so its size counts; + - implement `backend_deallocate` with interior mutability; + - expose arena options; + - document `:erlang.garbage_collect/1` and short-lived inference processes. + +### 3.8 Architecture and API (Responsibility-Driven Design) + +Background: `docs/architecture.md` sections 3 and 9. + +- [ ] **ARCH-01 (P2)** `Ortex.Model.run/2` builds and destructures `%Ortex.Backend{}` inside `%Nx.Tensor{}` (`model.ex:44-64`). Give `Ortex.Backend` that job (`wrap(ref, shape, type)`, `ref!(tensor)`). +- [ ] **ARCH-02 (P2)** `Ortex.Util.copy_ort_libs/0` mirrors Rustler and `ort` build internals. The native build should place everything the NIF needs, or link statically. +- [ ] **ARCH-03 (P2)** Execution-provider knowledge lives in five places: config, Cargo features, `constants.rs`, `map_eps` and docs. The native layer should own the catalogue and report it. +- [ ] **ARCH-04 (P3)** Elixir passes the output dtype to `concatenate` and the bit width to `to_binary`, though the tensors know both. Let Rust derive and validate them. +- [ ] **ARCH-05 (P1)** Model metadata leaks `ort`'s `Debug` format (BUG-01). Return structured data. +- [ ] **ARCH-06 (P2)** Give the NIF error contract one owner: a single boundary module converts native results to `{:ok, _}`/`{:error, %Ortex.Error{}}`, and public functions decide whether to raise. +- [ ] **ARCH-07 (P1) Options API.** + - `Ortex.load(path, opts)` takes keyword options. + - An `Ortex.SessionOptions` module validates them; it owns names, defaults, docs and the legacy positional mapping. + - A Rust `SessionOptions` (`NifMap`) applies them through a single `init/2` NIF. + - Keep `load(path, eps)` and `load(path, eps, opt)` working. + - This unifies PRs #50, #51 and #52 (section 5). +- [ ] **ARCH-08 (P2)** Named inputs and outputs (ONNX-06). +- [ ] **ARCH-09 (P3)** Telemetry events (ONNX-16). +- [ ] **ARCH-10 (P3)** A logging bridge from ONNX Runtime to Logger (ONNX-15). +- [ ] **ARCH-11 (P2)** Decide the backend policy. Issue #23 ("Implement argmax") was closed without code. Either document "storage only: transfer before compute" and point users to `Nx.backend_transfer/2`, or implement selected ops. + +### 3.9 Tests + +- [ ] **TEST-01 (P1)** `test/shape/concat_test.exs:6-19` maps every dtype except `s8` to `{:s, 16}`, so 10 of 12 dtypes are never exercised. Fix the table (PR #47). +- [ ] **TEST-02 (P1)** Turn every probe result in `docs/testing-and-ci.md` section 4 into a regression test. +- [ ] **TEST-03 (P1) Add tiny generated fixture models** (`onnx.helper`, no PyTorch): + - bool in and out; + - int64 and other integer outputs (these would have caught PR #46's integer corruption); + - outputs with different shapes, so declared output order is tested (issues #26 and #37); + - string, sequence and map outputs; + - several dynamic axes; + - external data; + - a `com.microsoft` contrib op (issue #32). +- [ ] **TEST-04 (P2)** A concurrency test (many tasks, one model) and a bounded-memory test, tagged slow. +- [ ] **TEST-05 (P2)** Rust unit tests (`cargo test`) for conversions and error paths. +- [ ] **TEST-06 (P3)** `reshape_test.exs` has a TODO saying it does not validate the Ortex path; assert on the backend and the data. +- [ ] **TEST-07 (P3)** The resnet50 tests never run in CI. Use a small committed model instead, or cache the export. +- [ ] **TEST-08 (P3)** `test/test_helper.exs:5` uses `IO.warn`, so `mix test --warnings-as-errors` fails after a passing run whenever resnet50 is missing. Use `IO.puts(:stderr, ...)`. + +### 3.10 CI + +- [ ] **CI-01 (P1)** `actions/checkout@v2` (`ci.yml:20, 37`) is deprecated, and macOS installs floating Homebrew Elixir/Erlang. Use current actions and `erlef/setup-beam` everywhere. +- [ ] **CI-02 (P1)** Build a matrix of floor and newest: Elixir 1.17 / OTP 25 to 27 and Elixir 1.20.4 / OTP 29.1, plus Nx 0.x and 1.0 while both are supported. +- [ ] **CI-03 (P2)** Cache `deps`, `_build`, `~/.cargo` and the ONNX Runtime download (BUILD-03). +- [ ] **CI-04 (P2)** Add `cargo fmt --check`, `cargo clippy -- -D warnings`, `mix docs`, `mix hex.build`, and a Rust beta build (BUILD-10). +- [ ] **CI-05 (P2)** Add Windows and Linux aarch64 jobs. +- [ ] **CI-06 (P2)** Fork PRs need maintainer approval to run: PR #52 is `action_required`. PR #51's runs failed with "workflow file issue" and zero jobs. Review the workflow settings. +- [ ] **CI-07 (P3)** Add Dependabot or Renovate for cargo, mix and GitHub Actions. + +### 3.11 Documentation + +- [ ] **DOC-01 (P1) README.** Fix: + - the `inspect` output; + - argmax now prints `s32`, not `s64` (Nx 0.9+ default integer type); + - installation requirements (Rust, network at build time, supported targets); + - GPU setup for applications (features in the app's own config, CUDA libraries); + - the silent provider fallback. +- [ ] **DOC-02 (P2) Module docs.** Fix `Ortex.Model`'s example output. In `Ortex.load`, cover the provider list, optimisation levels and `ORT_LIB_LOCATION`. +- [ ] **DOC-03 (P2)** Add `CHANGELOG.md` (PR #49 adds one). +- [ ] **DOC-04 (P3)** Replace `native/ortex/README.md` (the Rustler template, mentioning an `add/2` NIF) with a pointer to `docs/native-layer.md`. +- [ ] **DOC-05 (P3)** Ship the user-relevant parts of `docs/` (execution providers, building, Docker, Nerves) as ExDoc extras. +- [ ] **DOC-06 (P3)** The Python scripts write into the current directory instead of `models/` (PR #47 fixes `export_resnet.py`). `export_resnet.py` uses the deprecated `pretrained=True`. +- [ ] **DOC-07 (P3)** Document the supported opset and IR versions per release, and the external-data requirement (issues #16, #36). + +### 3.12 Repository hygiene + +- [ ] **REPO-01 (P3)** Delete the merged 2023 branches `develop`, `jv-readme` and `tm-backend-ops`. Keep `tm-execution-providers` (2024-03, provider option structs) or convert it into an issue. +- [ ] **REPO-02 (P3)** `mix.lock` has stale entries (axon, axon_onnx, protox, erlex, dll_loader_helper, useful, ...). Run `mix deps.unlock --unused`. +- [ ] **REPO-03 (P3)** Add a toolchain file (`.tool-versions` or `mise.toml`). +- [ ] **REPO-04 (P3)** Tags exist only for 0.1.8 to 0.1.10, and there are no GitHub releases. +- [ ] **REPO-05 (P2) Maintainer bandwidth.** + - There is one maintainer: `@mortont` (Travis Morton), the only Hex owner (`travismorton`). + - Last merge 2026-02-10; last maintainer comment 2026-02-24. + - No responses on PRs #49 to #52 or issue #27 since. + - About 2.7k Hex downloads a week. + - Consider adding maintainers or a second Hex owner. + +--- + +## 4. GitHub issues (25 total, reviewed 2026-10-29) + +| # | State | Title | Status on `main` today | Action | +| --- | --- | --- | --- | --- | +| 45 | open | Upgrade to ort 2.0.0-rc.10 | Blackwell GPUs fail on the ONNX Runtime 1.19 CUDA build. PR #46 (reject) and PR #47 (rc.11) are stalled waiting for `ort` 2.0. | Section 3.2; go to rc.15 now | +| 44 | open | Memory leak | No leak found in Ortex or `ort` rc.8 code paths; no unbounded growth reproduced. Likely GC-invisible native memory or arena growth. Reporter never sent a repro. | PERF-08; ask for a repro (Windows, YOLO, varying shapes) | +| 41 | open | Cross-compiling for Nerves | A community recipe works for aarch64. Blockers are listed in BUILD-04. | BUILD-04, ORT-14, BUILD-01 | +| 32 | open | Missing contrib operators | `MatMulNBits` and other `com.microsoft` ops are in the 1.19.2 binary. Probably fixed in 0.1.10; the reporter never re-tested. | TEST-03, then close | +| 29 | open | Session from memory | API available since rc.8. | ONNX-08 | +| 27 | open | Rustler precompiled | PR #49 open since 2026-04-26; no maintainer reply. | BUILD-02 | +| 43 | closed | Docker Debian compile | `cargo` was not on PATH. Rust is still required; rc.10+ will also need OpenSSL and rc.14+ Rust 1.92. | BUILD-02, ORT-11, DOC-05 | +| 42 | closed | BEAM crash loading a model | Workaround only (lower optimisation level). The stack overflow on a dirty IO thread is still live. | SAFE-05 | +| 37 | closed | silero VAD output order | Fixed in 0.1.10; no regression test. | TEST-03 | +| 36 | closed | Upgrade to 1.16 | ai.onnx.ml opset 4 works on 1.19; opset drift continues. | ONNX-01 | +| 34 | closed | Missing onnxruntime on Mac | Fixed in 0.1.10. Intel Mac support ends with the next upgrade. | ORT-10 | +| 33 | closed | More flexible rustler dependency | Done via a private-API version switch. | COMPAT-07 | +| 30 | closed | BERT example | Done (`examples/distilbert`). | none | +| 26 | closed | Unable to use ortex from main | Fixed in 0.1.10. The Windows DLL glob is still wrong. | BUILD-01, TEST-03 | +| 24 | closed | Update ort to 1.16.3 | Superseded. | Section 3.2 | +| 23 | closed | Implement argmax | Closed without implementation. | ARCH-11 | +| 20 | closed | Update to Nx ~> 0.6 | Done in 0.1.8. The same problem is back with Nx 1.0. | COMPAT-01 | +| 19 | closed | Bump Nx version | Duplicate of #20. | none | +| 17 | closed | Ort execution providers | The "fix" (shipping `config/config.exs`) has no effect for dependents; providers fall back silently. | BUG-03, BUILD-06, DOC-01 | +| 16 | closed | Error loading onnx model | Opset and export issue on the user side. | DOC-07 | +| 14 | closed | Rustler 0.29 for OTP 26 | Done; the crate is now 10 minor versions behind. | COMPAT-02 | +| 11 | closed | reshape, squeeze, concatenate | Done; full copies on normal schedulers. | BUG-10 | +| 8 | closed | slice | Done; tensor start indices unsupported. | BUG-06, BUG-10 | +| 6 | closed | Docs tidy up | Done. | none | +| 1 | closed | Building on macOS M2 | Fixed (per-OS features, rpath). | none | + +Maintainer statements worth knowing: + +- Precompiled NIFs and Ortex-managed ONNX Runtime binaries are "on the roadmap" (#27, #34, #17). +- From-memory loading was waiting on `ort` 2.0 (#29). +- They "would love" Nerves on rpi4 and Jetson (#41). +- They are waiting for `ort` 2.0 stable and have considered binding libonnxruntime directly (#45). +- They asked for PR #47 to be split (Feb 2026). + +--- + +## 5. Pull requests (reviewed statically 2026-10-29) + +| PR | Author | State and CI | What it does | Verdict | +| --- | --- | --- | --- | --- | +| #52 | omondiedward | open; CI never ran (fork approval pending) | Typed options: `:intra_op_num_threads`, `:inter_op_num_threads`, `:execution_mode`, spinning flags. Validated by a new `Ortex.SessionOptions`, applied through a second NIF `init_with_options/4`. | **Base for the consolidated options PR**, with changes: one NIF, keyword `load(path, opts)`, absorb #50, fix formatting (one line over 98 columns), drop `unless`, trim README, rebase on #47, credit #50 and #51. | +| #51 | pshoukry | open; CI never ran ("workflow file issue") | `intra_threads` option validated with `Keyword.validate!`; NIF `init/4`. | **Close in favour of the consolidated PR.** Keep its validation, tests and benchmark rationale. Its docs claim ONNX Runtime honours `ORT_INTRA_OP_NUM_THREADS`, which is only true from 1.29. | +| #50 | mattnenterprise | open; Linux CI failed (CDN 404, unrelated), macOS green | Raw ONNX Runtime config entries as a 4th positional argument; NIF `init/4`. | **Port as `:config_entries`**, applied before the typed options. Reject `nil` and malformed entries. Link the header for the bundled ONNX Runtime version (many keys only exist in 1.28+). | +| #49 | kipcole9 | open; CI green incl. 10-cell release matrix | `rustler_precompiled` for 5 targets x NIF 2.16/2.17; `ORTEX_BUILD` forces a source build; release workflow; `RELEASE.md`; `CHANGELOG.md`. | **Merge after changes, ideally after the `ort` upgrade.** Changes: generate the checksum file in the release flow (it ships as `%{}`); decide provider features per artifact (no CoreML/DirectML today); add GPU `variants` or document source builds; lower the glibc baseline; drop `x86_64-apple-darwin` for new ONNX Runtime; NIF 2.15 is enough; require `rustler_precompiled >= 0.10.1`; fix the OTP-range comment; rebase on #47 (5 conflicting files). | +| #47 | serpent213 | open; mergeable; CI green | `ort` rc.11 (ONNX Runtime 1.23), Rustler 0.37 with `resource_impl`, safe `from_binary`, panics turned into errors, input count check, `to_binary` limit, unknown providers rejected, multi-partition serving, concat test fix, CPU default on Linux, env switches, version 0.2.0-rc.1. | **Split it as the maintainer asked; fix before merging:** map level `3` to `All` (ORT-06); make `show_session` lock-free (ORT-12); avoid serialising inference (ORT-03); move `ORTEX_FEATURES` out of `config/config.exs` (BUILD-06); simplify `copy_ort_libs` (it globs the consumer project and `target/**`, has dead `cond` branches, and still misses Windows and triple directories); widen Nx to include 1.0; use plain `use Rustler`; then go to rc.15 (about four import fixes including `ort::tensor` to `ort::value`, provider `#[cfg]` gating, the one-per-process environment, `PreferCPU` for CPU-only sessions, `ORT_LIB_PATH`). | +| #46 | m0rt3nlund | open; conflicts; no CI | `ort` rc.10, Rustler 0.37, Nx 0.10, plus app-specific YOLO and image code. | **Close as superseded by #47.** It also corrupts integer outputs: unsigned outputs are clamped to `u8` and signed ones remapped (`tensor.rs:295-367`), and dtypes are reported as `{:u, 8}`. Also: CRLF rewrites of whole files, hard-enabled CUDA, `tracing_subscriber::fmt::init()` in `on_load`. Keep only the idea of an opt-in strict provider flag. | +| #28 | michaelbearne | closed 2024-11-19 | `ort` rc.2. | Superseded; nothing to port. | + +**Textual conflicts** (`git merge-tree`): + +- Each open PR merges cleanly on `main` except #46. +- #47 conflicts with #49 in 5 files. +- #50, #51 and #52 conflict with each other in the `load` path. +- #47 conflicts with #50 or #51 in one hunk of `lib.rs`. + +**Suggested merge order:** + +1. Close #46. +2. Phase 0 and phase 1 changes, including #47 part A. +3. #47 part B, retargeted to rc.15. +4. The consolidated options PR, then close #50, #51 and #52. +5. #49, rebased, with a version bump (the NIF arity changes). + +--- + +## 6. Verification log + +| What | Result | +| --- | --- | +| `main` on Elixir 1.20.4 / OTP 29.1.2 / Rust 1.99.1 (macOS arm64) | compile OK (dependency warnings only), format OK, 117 tests pass, 2 excluded | +| Fresh consumer app depending on `main` | resolves Nx 0.13.1 + Rustler 0.38.0; compiles; runs `tinymodel.onnx` | +| Dependency bump probe (Nx 1.0.0, Hex Rustler 0.38, ex_doc 0.40.4, crate unchanged) | compile, format, docs OK; 117 tests pass | +| Crate Rustler 0.39 probe | three layers of compile errors, fixed mechanically; full stack passes 117 tests on OTP 29 | +| Behaviour probe (`docs/testing-and-ci.md` section 4) | BUG-01 to BUG-06, BUG-14, BUG-15, SAFE-02 reproduced | +| Memory probe | bounded growth; no leak reproduced (PERF-08) | +| Rust 1.100.0-beta.7 (isolated toolchain) | static CPU build of `main` OK; dynamic ONNX Runtime build fails in `ort-sys` rc.8 `copy-dylibs` (BUILD-10); same build OK on 1.99.1 | +| `ort` source checks | rc.8 `Session` is `Send + Sync` with `run(&self)`; rc.15 `run(&mut self)`; `Level3` remapped; rc.8 `copy-dylibs` path logic | +| `ort` rc.15 compile probe (clone of `main`, only versions changed) | 41, then 43, 2 and 1 errors in successive waves; about 172 changed lines to a clean `cargo check` (also with `coreml` and `cuda,tensorrt` features) | +| `ort` rc.15 runtime checks | second `ort::init().build()` in one process fails; static environment + leaked sessions exit cleanly (3/3); unsatisfiable provider features still build a NIF on macOS that fails at load | +| PR builds | not executed (fork code); static review only | diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..5945b9b --- /dev/null +++ b/docs/README.md @@ -0,0 +1,62 @@ +# Ortex maintainer and agent docs + +These documents explain how Ortex is built and how it works inside, for people and AI agents who +change it. User-facing documentation stays in the top-level `README.md` and the module docs. + +Snapshot: 2026-10-29, `main` at `55dc3de` (Hex `0.1.10` + PR #48). Each document says which facts were +verified by running code and which come from reading it. + +## Reading order + +1. [`../AGENTS.md`](../AGENTS.md): setup, commands, gotchas, design rules, recipes. +2. [`architecture.md`](architecture.md): layers, responsibilities, the load/run/transfer/serving + flows, memory, concurrency, error model, invariants, design debts. +3. [`elixir-layer.md`](elixir-layer.md): public API behaviour, `Nx.Backend` coverage, dtype matrix, + configuration keys. +4. [`native-layer.md`](native-layer.md): the Rust crate: NIF table, types, conversions, unsafe and + panic inventories, the `ort` API surface in use. +5. [`build-and-packaging.md`](build-and-packaging.md): how `mix compile` builds the NIF, how ONNX + Runtime is downloaded and linked, platforms, environment variables, Hex package, precompiled + NIFs, cross-compiling. +6. [`execution-providers.md`](execution-providers.md): CPU/CUDA/TensorRT/CoreML/DirectML and friends, + how selection and fallback work, what changes upstream. +7. [`testing-and-ci.md`](testing-and-ci.md): running tests, test models, verified results, + behaviour probe results, CI. +8. [`upgrading.md`](upgrading.md): moving to the latest Elixir/Nx/Rustler and to the latest `ort` and + ONNX Runtime, with checklists and the order to do it in. +9. [`../TODO.md`](../TODO.md): every finding from the 2026-10 review, GitHub issue and PR triage, + and a roadmap. + +## Version snapshot + +| Component | Ortex `main` | Latest upstream (2026-10-29) | +| --- | --- | --- | +| Elixir / Erlang/OTP | declares `~> 1.14`; CI 1.17.3 / 27.1.2 | 1.20.4 / 29.1.2 (verified working) | +| Nx | `~> 0.6` (lock 0.6.2) | 1.0.0 | +| Rustler (Hex / crate) | `~> 0.27` (lock 0.29.1) / `0.29` | 0.38.0 / 0.39.0 | +| rustler_precompiled | not used | 0.10.1 | +| `ort` crate | `2.0.0-rc.8` | `2.0.0-rc.15` (no 2.0.0 yet) | +| ONNX Runtime | 1.19.2 (via `ort`) | 1.31.1 (`ort` rc.15 bundles 1.31.0) | +| Prebuilt ONNX Runtime platform floor | about glibc 2.32; Intel and Apple Silicon macOS | glibc 2.39 / libstdc++ 13.2, AVX2, Apple Silicon only (rc.15) | +| ONNX opset supported | 21 (ONNX 1.16.1) | 26 in ONNX Runtime 1.28 to 1.31 (ONNX 1.22); ONNX itself is at 1.23.2 / opset 28 | +| `ndarray` / `half` | 0.16.1 / 2.2.1 | 0.17.2 / 2.7.1 | +| ex_doc | pinned `0.29.4` | 0.40.4 | + +## Glossary + +| Term | Meaning | +| --- | --- | +| ONNX | Open Neural Network Exchange: a file format (protobuf) for ML models. Versioned by *IR version* (file format) and *opset* (operator set) versions. | +| ONNX Runtime (ORT) | Microsoft's C++ inference engine for ONNX models. | +| `ort` | The Rust crate (by pyke, `pykeio/ort`) that binds ONNX Runtime. Its 2.0 line has been in release candidates since 2024. | +| pyke CDN | Where `ort` downloads prebuilt ONNX Runtime builds during `cargo build` (`parcel.pyke.io` for rc.8, `cdn.pyke.io` later). | +| Execution provider (EP) | An ONNX Runtime backend for some hardware (CPU, CUDA, TensorRT, CoreML, DirectML, ...). | +| Session | A loaded, optimised model in ONNX Runtime, ready to run. Ortex wraps one per `%Ortex.Model{}`. | +| NIF | Native Implemented Function: a C-ABI function loaded into the BEAM. Ortex's NIFs are written in Rust with Rustler. | +| Dirty scheduler | BEAM thread pool for NIFs that run longer than about 1 ms (dirty CPU, dirty IO). | +| Resource (`ResourceArc`) | A reference-counted native object owned by the BEAM GC. `OrtexModel` and `OrtexTensor` are resources. | +| Resource binary | A BEAM binary that points into a resource's memory without copying (`ResourceArc::make_binary`). | +| `Nx.Backend` | The behaviour that stores and computes tensors for Nx. `Ortex.Backend` only stores. | +| `Nx.Serving` | Nx's batching and serving abstraction; `Ortex.Serving` implements its callbacks. | +| rustler_precompiled | Library that downloads prebuilt NIFs from GitHub releases instead of compiling Rust. | +| NIF version | ABI version of the NIF API (OTP 24-25 = 2.16, OTP 26-28 = 2.17, OTP 29 = 2.18). A library built for an older minor version loads on newer runtimes. | diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..f27e4f8 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,274 @@ +# Ortex architecture + +Snapshot: `main` at `55dc3de` (2026-02-10), Hex release `0.1.10`. Everything below describes the +code as it is on `main` unless a section says otherwise. Planned changes live in +[`../TODO.md`](../TODO.md); upgrade work is described in [`upgrading.md`](upgrading.md). + +## 1. What Ortex is + +Ortex lets Elixir programs run ONNX models. It has three jobs: + +1. Load an `.onnx` file into an ONNX Runtime *session* and keep it alive inside the BEAM. +2. Move tensor data between [Nx](https://hexdocs.pm/nx) and ONNX Runtime and run inference. +3. Plug into `Nx.Serving` so a model can be batched and served from a supervision tree. + +It does not compile or train models and it does not implement Nx math. `Ortex.Backend` is a +*storage* backend: it holds tensors that came out of (or are about to go into) ONNX Runtime and +supports a handful of shape operations. + +## 2. Layers + +```mermaid +flowchart TD + App["Elixir application"] --> Facade["Ortex (lib/ortex.ex)"] + App --> Serving["Ortex.Serving (Nx.Serving behaviour)"] + App --> NxAPI["Nx API (Nx.backend_transfer, Nx.slice, ...)"] + Facade --> Model["Ortex.Model (struct + Inspect)"] + Serving --> Facade + NxAPI --> Backend["Ortex.Backend (Nx.Backend behaviour)"] + Model --> Backend + Model --> Native["Ortex.Native (NIF stubs, compile-time build)"] + Backend --> Native + Native -. "@on_load: priv/native NIF library" .-> Crate["Rust crate native/ortex (Rustler NIFs)"] + Crate --> Ort["ort crate 2.0.0-rc.8"] + Ort --> ORT["ONNX Runtime 1.19.2 (C++), static or dynamic"] + ORT --> EPs["Execution providers: CPU, CoreML, CUDA, TensorRT, DirectML, ..."] +``` + +| Layer | Files | Language | +| --- | --- | --- | +| Public API | `lib/ortex.ex`, `lib/ortex/model.ex`, `lib/ortex/serving.ex`, `lib/ortex/backend.ex` | Elixir | +| NIF boundary (Elixir side) | `lib/ortex/native.ex`, `lib/ortex/util.ex` | Elixir | +| NIF boundary (Rust side) | `native/ortex/src/lib.rs` | Rust | +| Native domain | `native/ortex/src/{model,tensor,utils,constants}.rs` | Rust | +| Inference engine | `ort` crate (Rust bindings) + ONNX Runtime (C++ library downloaded at build time) | Rust/C++ | + +Naming used in all docs: **`ort`** is the Rust crate by pyke (`pykeio/ort`); **ONNX Runtime** +(sometimes "ORT") is Microsoft's C++ engine that `ort` binds to. They have unrelated version +numbers: `ort 2.0.0-rc.8` bundles ONNX Runtime `1.19.2`. + +## 3. Components, responsibilities, collaborators + +The project follows Responsibility-Driven Design: each component owns its data and behaviour, +and collaborators ask it to act instead of reaching into its state. This table records what each +component knows and does today. Section 9 lists where today's code breaks that rule. + +### Elixir + +| Component | Knows | Does | Collaborators | +| --- | --- | --- | --- | +| `Ortex` | nothing (facade) | Delegates `load/3` and `run/2` to `Ortex.Model` | `Ortex.Model` | +| `Ortex.Model` | The NIF resource reference of one loaded session (`%Ortex.Model{reference: ref}`) | Loads a model (raises on `{:error, msg}`); runs inference: transfers inputs to `Ortex.Backend`, calls the NIF, wraps outputs as `Nx.Tensor`s; renders itself in `inspect` | `Ortex.Native`, `Ortex.Backend`, `Nx` | +| `Ortex.Backend` | The NIF resource reference of one tensor (`%Ortex.Backend{ref: ref}`) | Implements 9 `Nx.Backend` callbacks (`init`, `from_binary`, `to_binary`, `backend_transfer`, `inspect`, `slice`, `reshape`, `squeeze`, `concatenate`); every other callback is generated at compile time to raise `"operation X is not yet supported on Ortex.Backend."` | `Ortex.Native`, `Nx` | +| `Ortex.Serving` | The model closure | `Nx.Serving` callbacks: `init/3` builds `fn x -> Ortex.run(model, x) end`; `handle_batch/3` materialises the `Nx.Batch` with `Nx.Defn.jit_apply(&Function.identity/1, [batch])` and runs it | `Ortex`, `Nx.Serving`, `Nx.Defn` | +| `Ortex.Native` | NIF function names and arities | At **compile time**: builds the Rust crate (`Rustler.Compiler.compile_crate`), calls `Ortex.Util.copy_ort_libs/0`, then `use Rustler, skip_compilation?: true`. At **load time**: `@on_load` loads `priv/native/libortex.so`. Defines `:erlang.nif_error(:nif_not_loaded)` stubs | Rustler, cargo, `Ortex.Util` | +| `Ortex.Util` | Where cargo leaves `libonnxruntime*` files (`_build//lib/ortex/native/ortex/{release,debug}`) and their per-OS file names | Copies those files into `priv/native` so the NIF's rpath (`$ORIGIN`, `@loader_path`) finds them | file system | +| `Inspect` for `Ortex.Model` | nothing | Calls `Ortex.Native.show_session/1` and pretty-prints `{name, type_string, dims}` lists | `Ortex.Native` | + +### Rust (`native/ortex/src`) + +| Component | Knows | Does | Collaborators | +| --- | --- | --- | --- | +| `lib.rs` | The NIF table and each NIF's scheduler | Decodes BEAM terms, calls domain functions, encodes results or `{:error, String}`; registers the two resource types in the `load` callback | `model`, `tensor`, `utils`, Rustler | +| `model.rs` (`OrtexModel`) | One `ort::Session` | `init`: builds the session (optimisation level, execution providers, file path). `show`: lists inputs/outputs. `run`: converts inputs, runs the session, converts outputs | `ort`, `tensor`, `utils` | +| `tensor.rs` (`OrtexTensor`) | One owned `ndarray::ArrayD` per dtype variant (`s8..s64`, `u8..u64`, `f16`, `bf16`, `f32`, `f64`, internal `bool`) | Shape, dtype, raw bytes, slice, reshape, concatenate, u8-to-bool conversion, conversions to and from `ort` values | `ndarray`, `half`, `ort` | +| `utils.rs` | The dtype string/bit-width table for decoding Nx binaries; the atom-to-execution-provider table; the int-to-optimisation-level table | `from_binary`, `to_binary`, `map_eps`, `map_opt_level`, `is_bool_input` | `tensor`, `constants`, `ort` | +| `constants.rs` | Execution-provider name strings and the `rustler::atoms!` list | (data only) | | + +## 4. Key flows + +### 4.1 Build (happens inside `mix compile`) + +```mermaid +sequenceDiagram + participant Mix + participant Native as Ortex.Native (module body) + participant Rustler as Rustler.Compiler + participant Cargo + participant OrtSys as ort-sys build.rs + participant Util as Ortex.Util + Mix->>Native: compile lib/ortex/native.ex + Native->>Rustler: compile_crate(...) (API differs for Rustler < 0.30 and >= 0.30) + Rustler->>Cargo: cargo rustc --release (features from config :ortex, Ortex.Native) + Cargo->>OrtSys: run build script + OrtSys->>OrtSys: download ONNX Runtime 1.19.2 from parcel.pyke.io (or use ORT_LIB_LOCATION) + OrtSys-->>Cargo: link static libonnxruntime.a (CPU builds) or libonnxruntime.so/.dll (CUDA builds) + Cargo-->>Rustler: libortex.{so,dylib,dll} + Rustler-->>Native: copies NIF to priv/native (libortex.so with Rustler 0.29, ortex.so with 0.38) + Native->>Util: copy_ort_libs() copies libonnxruntime* (if any) into priv/native + Native->>Native: use Rustler, skip_compilation?: true (sets @on_load) +``` + +Details, platform matrix and environment variables: [`build-and-packaging.md`](build-and-packaging.md). + +### 4.2 Load + +`Ortex.load(path, eps \\ [:cpu], opt \\ 3)` +→ `Ortex.Model.load/3` +→ `Ortex.Native.init(path, eps, opt)` (dirty IO scheduler) +→ `utils::map_eps` turns atoms into `ExecutionProviderDispatch` values (unknown atoms silently become CPU) +→ `model::init` builds `Session::builder()?.with_optimization_level(..)?.with_execution_providers(eps)?.commit_from_file(path)?` +→ returns `ResourceArc`; Elixir wraps it in `%Ortex.Model{reference: ref}`. +Errors come back as `{:error, message}` and `Ortex.Model.load/3` raises `RuntimeError` with the message. + +### 4.3 Run + +```mermaid +sequenceDiagram + participant Caller + participant Model as Ortex.Model.run/2 + participant Backend as Ortex.Backend + participant NIF as Ortex.Native (Rust) + participant ORT as ONNX Runtime + Caller->>Model: run(model, {t1, t2}) + Model->>Backend: Nx.backend_transfer(t, Ortex.Backend) per input + Backend->>NIF: from_binary(binary, shape, type) (dirty CPU, copy 1) + NIF-->>Backend: ResourceArc + Model->>NIF: run(model_ref, [tensor_refs]) (dirty IO) + NIF->>NIF: zip inputs with session inputs by position, u8 to bool where the model wants bool + NIF->>NIF: clone each ndarray into an ort Value (copy 2) + NIF->>ORT: session.run(inputs) + ORT-->>NIF: outputs (ONNX Runtime-owned memory) + NIF->>NIF: try_extract_tensor + into_owned per output (copy 3); bool outputs become u8 + NIF-->>Model: [{tensor_ref, shape, dtype_atom, bits}] in model output order + Model-->>Caller: tuple of Nx.Tensor with data %Ortex.Backend{ref: ...} +``` + +Inputs are matched to model inputs **by position**. Extra inputs are silently dropped by `zip`; +missing inputs fail inside ONNX Runtime with an unhelpful kernel error (verified, see +[`testing-and-ci.md`](testing-and-ci.md#4-behaviour-probe-results)). All model outputs are always +computed and returned. + +### 4.4 Getting data back to Elixir + +`Nx.backend_transfer(t)` → `Ortex.Backend.backend_transfer/3` → `Ortex.Native.to_binary(ref, 0, 0)` +returns a **resource binary** created with `ResourceArc::make_binary`: the BEAM binary points +straight at the tensor's memory (no copy) and keeps the `OrtexTensor` alive for as long as the +binary, or any sub-binary of it, is referenced. `Nx.BinaryBackend.from_binary/3` stores that +binary as-is. + +### 4.5 Serving + +`Nx.Serving.new(Ortex.Serving, model)` → `init/3` returns a closure over the model. +`handle_batch/3` turns the `Nx.Batch` into a tensor tuple by running +`Nx.Defn.jit_apply(&Function.identity/1, [batch])` (stacking happens on the default backend, +normally `Nx.BinaryBackend`), then calls `Ortex.run/2` inside the serving process and returns +`{:execute, fn -> {output, :server_info} end, state}`. `init/3` pattern-matches exactly one +partition (`[_defn_options]`), so `partitions: true` with more than one partition crashes. + +## 5. Memory and lifetimes + +* Both NIF resources (`OrtexModel`, `OrtexTensor`) are reference counted by the BEAM and freed + when the last Elixir reference (term or resource binary) is garbage collected. +* The BEAM only sees a small reference term for a resource, not the megabytes behind it. A + process that creates many large tensors without growing its own heap may not garbage collect + for a long time, so native memory can climb. This is the usual first suspect for reports like + issue #44 ("Memory leak"). Resource binaries returned by `to_binary` *are* counted as off-heap + binary memory, so they trigger GC normally. +* Copies per `Ortex.run/2` today: one copy into Rust when an input comes from another backend, + one clone into an `ort` value for every input (three for bool inputs), and one copy out of + ONNX Runtime for every output. Section 4.4's transfer back to Elixir is zero-copy. +* Every `OrtexTensor` array must be in standard (row-major, contiguous) layout because + `to_bytes` reinterprets the array's memory as a byte slice. Every code path that creates an + array today produces standard layout (`concatenate` rebuilds explicitly). Keep it that way. +* Tensors are immutable once created. Nothing may mutate an `OrtexTensor` in place, because + resource binaries alias its memory. + +## 6. Concurrency and scheduling + +| NIF | Scheduler | Notes | +| --- | --- | --- | +| `init/3` | dirty IO | Reads the file and optimises the graph. | +| `run/2` | dirty IO | Blocks while ONNX Runtime's own thread pool computes. | +| `from_binary/3`, `to_binary/3` | dirty CPU | Copies or aliases tensor memory. | +| `show_session/1` | normal | Cheap metadata read. | +| `slice/4`, `reshape/2`, `concatenate/3` | normal | **Copy whole arrays**; large tensors can exceed the 1 ms budget of a normal scheduler. Should be dirty CPU. | + +* One `OrtexModel` is shared by every process that holds the `%Ortex.Model{}`. With + `ort 2.0.0-rc.8`, `Session::run` takes `&self`, ONNX Runtime's `Run` is thread safe, and the + crate adds `unsafe impl Sync for OrtexModel {}`. Concurrent `Ortex.run/2` calls on one model + run in parallel (verified with 32 concurrent tasks). +* That parallelism is only safe on some providers. `ort`'s maintainers made `Session::run` take + `&mut self` (rc.10+) because concurrent inference on one session "often saw crashes and memory + corruption", and ONNX Runtime forbids concurrent `Run()` on one session with DirectML and CUDA + graphs. Open PR #47 answers that with a `Mutex`, which serialises inference per model; see + [`upgrading.md`](upgrading.md) section 3.3 for the options (mutex, session pool, per-partition + sessions). +* After the upgrade there is exactly one ONNX Runtime `Environment` per OS process (a second + `ort::init().build()` fails), owned by the NIF and shared by every session. +* Each session gets its own ONNX Runtime intra-op thread pool sized to the machine by default (with + busy-wait spinning on in rc.8; pyke's builds from rc.11 turn spinning off by default). Loading + several models therefore creates several pools that compete with each other and with BEAM + schedulers. Ortex exposes no thread or spinning settings today (open PRs #50, #51, #52); after the + upgrade a process-wide global thread pool becomes possible. + +## 7. Error model + +| Source | What the caller sees | +| --- | --- | +| `ort`/ONNX Runtime error, invalid shape, wrong dtype, missing file | NIF returns `{:error, message}` (`rustler::Error::Term`). `Ortex.Model` and `Ortex.Backend.from_binary/to_binary` raise `RuntimeError, message`. | +| `{:error, _}` from `slice`, `reshape`, `concatenate` | **Not checked** by `Ortex.Backend` on `main`; the error tuple would be stored as the tensor's `ref`. | +| Term decoding failure (wrong argument type, e.g. a string execution provider or a tensor start index in `Nx.slice`) | `ArgumentError` ("argument error"). | +| Rust panic (`unimplemented!`, `todo!`, `panic!`, `unwrap`) | Rustler catches it: `ErlangError` with reason `:nif_panicked`, plus a panic message on stderr. The VM keeps running. | +| Unsupported Nx operation on an Ortex tensor | `RuntimeError, "operation is not yet supported on Ortex.Backend."` | + +## 8. Invariants (keep these true) + +1. The set of NIFs is declared three times and must match: `#[rustler::nif]` functions in + `lib.rs`, the list in `rustler::init!` (required by Rustler 0.29), and the stub functions in + `lib/ortex/native.ex` (same names and arities). +2. `OrtexTensor` arrays are in standard layout and never mutated after creation (section 5). +3. The bool variant never escapes to Elixir. Nx has no boolean type: ONNX `bool` outputs become + `{:u, 8}` and `{:u, 8}` inputs are converted to bool only when the model input is bool + (values must be 0 or 1). +4. Anything that may take longer than about 1 ms runs on a dirty scheduler. +5. NIF code paths reachable with user data must return errors, not panic. +6. Output order equals `session.outputs` order; input order equals `session.inputs` order. + +## 9. Responsibility problems in today's code + +Each item names the component that should own the knowledge; the fixes are tracked in +`TODO.md` (section "Architecture and API"). + +1. **`Ortex.Model` builds and takes apart `Ortex.Backend` data.** `Model.run/2` pattern-matches + `%Nx.Tensor{data: %Ortex.Backend{ref: x}}` and constructs `%Nx.Tensor{data: %Ortex.Backend{...}}` + by hand. The backend should own that: e.g. `Ortex.Backend.to_ref/1` and + `Ortex.Backend.wrap(ref, shape, type)`. +2. **`Ortex.Util.copy_ort_libs/0` mirrors the build system.** It hard-codes Rustler's target + directory layout, cargo profile folders, `ort`'s download behaviour and per-OS library names. + It is already broken for most users: Hex resolves `{:rustler, "~> 0.27"}` to Rustler 0.38, which + builds into `deps/ortex/native/ortex/target/release/` (verified), while the function only looks in + `_build//lib/ortex/native/ortex/{release,debug}` (the Rustler 0.29 layout). It also misses + Windows (`onnxruntime.dll` has no `lib` prefix). The build should produce a self-contained NIF + (static linking, or the native build placing its own side libraries) so Elixir needs no + knowledge of it. +3. **Execution-provider knowledge is spread across five places**: `config/config.exs` (which + Cargo features per OS), `Cargo.toml` features, `constants.rs`, `utils::map_eps` (with a silent + CPU fallback), and the docs. Elixir cannot ask what was compiled in or is usable. The native + layer should own the catalogue and answer `available_execution_providers/0`, and loading + should fail (or warn) when a requested provider is unavailable. +4. **Elixir tells Rust facts the tensor already knows.** `concatenate/3` receives the output + type and `to_binary/3` receives the element bit width from Elixir, although every + `OrtexTensor` knows its dtype. Rust should derive them (and validate that concatenated inputs + agree); Elixir should pass only what it owns (axis, element limit). +5. **Model metadata leaks `ort`'s `Debug` format.** `model::show` returns + `format!("{:#?}", value_type)`, so the public `inspect` output changes whenever `ort` changes + its `Debug` impl (it already did: the README promises `"Float32"` and `nil` dimensions; the + code prints a multi-line `Tensor { ty: Float32, dimensions: [-1, ...] }` and `-1`). Rust + should return structured data (name, Nx type, dimensions with `nil` for dynamic axes, + symbolic dimension names); Elixir should format it. +6. **No single owner of the NIF error contract.** Some callers raise on `{:error, _}`, some + ignore it, some get `ArgumentError` from decoding. One Elixir module (the native boundary) + should own conversion of native results into `{:ok, _} | {:error, %Ortex.Error{}}`, and the + public functions should decide whether to raise. + +## 10. Design decisions and their reasons + +| Decision | Reason | Revisit when | +| --- | --- | --- | +| Storage-only `Nx.Backend` | ONNX Runtime does the math; tensors only need to cross the boundary and be reshaped/sliced for batching. | Never for math; implement the remaining *data movement* callbacks (`backend_copy`, `backend_deallocate`, `as_type`, `to_batched`). | +| Build the crate manually before `use Rustler` | `ort` downloads ONNX Runtime during cargo build; the side libraries must be copied into `priv/native` before the NIF is loaded, which `use Rustler` alone did not allow. | Static linking everywhere, or precompiled NIFs (PR #49). | +| Positional inputs, all outputs | Simple tuple API mirroring `Nx.Serving` containers. | Named inputs/outputs are a common request; see TODO. | +| Bool as `u8` | Nx has no boolean type; Nx predicates return `u8`. | If Nx adds a boolean type. | +| `run` on dirty IO | ONNX Runtime does the CPU work on its own intra-op threads while the calling BEAM thread mostly waits; the dirty IO pool (10 threads by default) is separate from the dirty CPU pool, so long inferences do not starve other CPU-bound NIFs. | If `run` becomes CPU-bound on the calling thread (e.g. intra-op threads set to 1), switch to dirty CPU. | +| Execution providers chosen per OS in `config/config.exs` | Convenience for developing Ortex itself. | It does not apply to dependents at all (Mix does not load a dependency's config); see `build-and-packaging.md`. | diff --git a/docs/build-and-packaging.md b/docs/build-and-packaging.md new file mode 100644 index 0000000..ef7d791 --- /dev/null +++ b/docs/build-and-packaging.md @@ -0,0 +1,236 @@ +# Build, linking and packaging + +Snapshot: `main` at `55dc3de` (`ort 2.0.0-rc.8`, ONNX Runtime 1.19.2). For what changes with newer `ort` +releases see [`upgrading.md`](upgrading.md). + +## 1. The short version + +* Ortex has **no precompiled NIF**. Every install compiles the Rust crate, so every user needs a + Rust toolchain (issue #27; PR #49 proposes `rustler_precompiled`). +* During that cargo build, `ort-sys`'s build script **downloads a prebuilt ONNX Runtime** from pyke's + CDN (`parcel.pyke.io`), verifies its SHA-256, and links it. CPU builds are linked **statically** + (no side library); CUDA builds are **dynamic** and need `libonnxruntime*` next to the NIF. +* Only six targets have prebuilt ONNX Runtime binaries. Anything else (Nerves armv7/armv6, musl, + RISC-V, FreeBSD) fails the build unless you supply ONNX Runtime yourself via `ORT_LIB_LOCATION`. + +## 2. What happens during `mix compile` + +The body of `lib/ortex/native.ex` runs at compile time: + +1. Reads the Rustler version (`Application.spec(:rustler, :vsn)`) and calls the matching API: + `Rustler.Compiler.compile_crate(:ortex, Application.compile_env(:ortex, Ortex.Native, []), otp_app: :ortex, crate: :ortex)` + for Rustler >= 0.30, or `compile_crate(__MODULE__, otp_app: :ortex, crate: :ortex)` for older ones. +2. Rustler runs cargo in release mode with the configured Cargo features (observed: "Compiling crate + ortex in release mode" even with `MIX_ENV=test`). +3. `ort-sys`'s build script picks a binary *variant* from the enabled features, downloads and links it + (section 3). +4. Rustler copies the built NIF into `priv/native/`. File name and build directory depend on Rustler: + + | Rustler (Hex) | Cargo target dir | NIF location | + | --- | --- | --- | + | 0.29.x to 0.36.x (this repo's `mix.lock` has 0.29.1) | `_build//lib/ortex/native/ortex//` (Rustler sets `CARGO_TARGET_DIR`) | `_build//lib/ortex/priv/native/libortex.so` | + | 0.37.0+ (`~> 0.27` resolves to 0.38.0 for new users) | cargo's default: `/native/ortex/target//` (for a dependency: `deps/ortex/native/ortex/target/release/`) | `/priv/native/ortex.so`; `_build//lib/ortex/priv` becomes a symlink to the source `priv/` | + + The 0.37.0 change is not in Rustler's changelog ("Refactor build logic to remove TOML dependency"); + it was confirmed by comparing the 0.36.2 and 0.37.0 sources and by building a consumer app. + Because the NIF now lands in the source tree, `/priv/native/` should be in `.gitignore`. + +5. `Ortex.Util.copy_ort_libs/0` globs `libonnxruntime*.{so,dylib,dll}*` in + `_build//lib/ortex/native/ortex/{release|debug}` and copies matches into `priv/native/`. + **With Rustler >= 0.37 that directory does not exist**, so nothing is copied. Static builds do not + care; dynamic (CUDA) builds then fail to load with "cannot open shared object file". On Windows + the glob also misses `onnxruntime.dll` (no `lib` prefix), and cross-compiled builds put their + output under `target///`, which is not searched either. +6. `use Rustler, otp_app: :ortex, crate: :ortex, skip_compilation?: true` installs `@on_load`, which + loads the NIF from `priv/native/` when the module loads. + +Mix does not track the Rust sources (no `@external_resource`), so editing them does not trigger a +rebuild: use `mix compile --force`. The cause is `skip_compilation?: true`: Rustler only records the +crate's files as external resources when it compiles the crate itself. Replacing the private +`Rustler.Compiler.compile_crate` call with a plain `use Rustler, otp_app: :ortex, crate: :ortex` +followed by the library copy works with Rustler 0.38 and restores rebuild tracking (verified). + +### Rust 1.100 and `copy-dylibs` (deadline about 2026-11-12) + +Rust 1.100 ships cargo's new build-dir layout (`OUT_DIR` moves one level deeper, to +`target//build///out`). `ort-sys` rc.8 finds the target directory with +`OUT_DIR.ancestors().nth(3)` and `unwrap()`s symlink creation there. On Rust 1.100.0-beta.7: + +* static CPU builds (no dylibs to copy) build fine; +* any build whose ONNX Runtime is a dynamic library (the `cu12` CUDA/TensorRT download, or a + dynamic `ORT_LIB_LOCATION`) fails: `failed to run custom build command for ort-sys v2.0.0-rc.8 ... + panicked at build.rs:111:66 ... No such file or directory`. + +`ort` rc.14 and later support the new layout. Until Ortex upgrades, GPU users must stay on Rust 1.99, +and this repository's Linux CI (which enables `cuda`/`tensorrt`) will fail once runners update. + +## 3. How `ort` 2.0.0-rc.8 obtains ONNX Runtime + +Default `ort` features: `ndarray`, `half`, `download-binaries`, `copy-dylibs`. + +**Variant selection** (`ort-sys` build script): `train` if the `training` feature is on; `cu12` if +`cuda` or `tensorrt` is on (else `rocm` if `rocm`); otherwise `none`. If the variant does not exist for +the target, it silently falls back to `none`; if `none` does not exist either, the build panics with +"downloaded binaries not available for target ... you may have to compile ONNX Runtime from source". + +| Target | `none` (CPU, static) | `cu12` (CUDA 12, dynamic) | `train` | +| --- | --- | --- | --- | +| `aarch64-apple-darwin` | yes (includes CoreML) | no | yes | +| `x86_64-apple-darwin` | yes | no | yes | +| `x86_64-unknown-linux-gnu` | yes | **yes** | yes | +| `aarch64-unknown-linux-gnu` | yes | no (falls back to CPU) | yes | +| `x86_64-pc-windows-msvc` | yes | **yes** | yes | +| `aarch64-pc-windows-msvc` | yes | no | yes | +| anything else (armv7, musl, riscv64, ...) | **no: build fails** | no | no | + +Binaries come from `https://parcel.pyke.io/v2/delivery/ortrs/packages/msort-binary/1.19.2/...` +(still reachable on 2026-10-29) and are checked against SHA-256 values compiled into `ort-sys`. + +**Environment variables** read by `ort-sys` rc.8: + +| Variable | Effect | +| --- | --- | +| `ORT_LIB_LOCATION` | Use an ONNX Runtime you built or downloaded instead of downloading. Points at the directory with the libraries (static or dynamic). | +| `ORT_LIB_PROFILE` | Build profile subdirectory to use inside an ONNX Runtime build tree. | +| `ORT_PREFER_DYNAMIC_LINK` | Link dynamically even if a static library is available. | + +`copy-dylibs` copies dynamic libraries into cargo's output directory so something can place them +next to the NIF (that something is `Ortex.Util.copy_ort_libs/0`). The `load-dynamic` feature (load +ONNX Runtime at run time from `ORT_DYLIB_PATH`) is not used. + +## 4. Linking and loading the NIF + +`native/ortex/.cargo/config.toml` (also shipped in the Hex package): + +| Setting | Why | +| --- | --- | +| `rpath = true` for dev and release profiles | Let the NIF find side libraries relative to itself. | +| macOS: `-undefined dynamic_lookup` | NIF symbols (`enif_*`) are resolved from the VM at load time. | +| macOS: `-fapple-link-rtlib` | The static ONNX Runtime needs compiler-rt builtins (e.g. `___isPlatformVersionAtLeast`); added by the "fix for MacOS linking failure" commit. | +| macOS: `-Wl,-rpath,@loader_path`; Linux x86_64/aarch64: `-Wl,-rpath,$ORIGIN` | Search the NIF's own directory (`priv/native`) for `libonnxruntime*`. | + +Windows has no rpath. A dynamically linked ONNX Runtime must be found through the DLL search order +(directory of the executable, system directories, `PATH`), which `priv/native` is not part of. + +With the Rustler crate 0.36 or later, `enif_*` symbols are resolved at load time by Rustler itself, +so macOS no longer needs `-undefined dynamic_lookup`. A probe with crate 0.39 linked and passed all +tests on macOS arm64 with only `-Wl,-rpath,@loader_path`, without `-fapple-link-rtlib` either. With +crate 0.29 (today), removing `-undefined dynamic_lookup` fails with "Undefined symbols _enif_*". + +Verified on macOS arm64: the NIF is a single 21 MB library linking ONNX Runtime statically, plus +system frameworks (Foundation, CoreFoundation, CoreML) and `libc++`. + +## 5. Choosing Cargo features (execution providers) + +* **This repository** (`config/config.exs`, dev and test only): Windows `["directml"]`, macOS + `["coreml"]`, other Unix `["cuda", "tensorrt"]`. On Linux this makes CI download the CUDA 12 + dynamic build. +* **Applications using Ortex**: Mix never loads a dependency's `config/config.exs`, so the default is + **no features**: the static CPU build on every target. To get GPU support an application sets: + + ```elixir + # config/config.exs of the application + config :ortex, Ortex.Native, features: ["cuda"] + ``` + + and then needs the CUDA 12 and cuDNN libraries at run time, plus a working `copy_ort_libs` + (broken with Rustler >= 0.37, see section 2). The `coreml` and `directml` features only matter + for registering those providers; on rc.8 their binaries are the same static `none` build. + +Details per provider: [`execution-providers.md`](execution-providers.md). + +## 6. The Hex package + +`mix.exs` `package[:files]`: `lib`, `.formatter.exs`, `mix.exs`, `README*`, `LICENSE*`, +`native/ortex/src/`, `config/config.exs`, `native/ortex/Cargo.lock`, `native/ortex/Cargo.toml`, +`native/ortex/.cargo/config.toml` (matches the published `0.1.10` tarball). Shipping +`config/config.exs` has no effect on users. Published requirements: `rustler ~> 0.27`, `nx ~> 0.6`. + +Because the requirements are loose, a new application today resolves **Rustler 0.38.0 and Nx 0.13.1** +(not the versions in this repo's `mix.lock`). Verified on Elixir 1.20.4 / OTP 29.1.2 / macOS arm64: +that combination compiles (Rust warnings only) and runs `models/tinymodel.onnx`. It cannot be +combined with anything that requires Nx 1.0. + +## 7. Releasing (current practice) + +There is no release automation and no `CHANGELOG`. Releases so far: bump `version` in `mix.exs`, tag +`vX.Y.Z` (tags exist only for 0.1.8 to 0.1.10), `mix hex.publish` by the Hex owner (`travismorton`, +GitHub `@mortont`). PR #49 proposes a tag-driven workflow that builds NIFs for 5 targets x NIF 2.16/2.17 +and a `RELEASE.md` runbook. + +## 8. Precompiled NIFs (not yet available) + +Requested in issue #27; implemented in open PR #49 with `rustler_precompiled`. Things to settle +before shipping precompiled artifacts: + +* Artifacts must be self-contained. Static ONNX Runtime (the `none` variant) is; GPU variants are not. + GPU builds need either `rustler_precompiled` *variants* that ship the ONNX Runtime side libraries, + or remain source-build only. +* Which providers each artifact supports must be documented: PR #49 builds without Cargo features, + so CoreML (macOS) and DirectML (Windows) providers would not register in precompiled builds. +* NIF versions (from `erl_nif.h` at each OTP tag): OTP 24 and 25 are 2.16; OTP 26, 27 and 28 are 2.17; + OTP 29 is 2.18. A library built for a lower minor version loads on a newer runtime, so 2.16 and 2.17 + artifacts cover OTP 24 to 29. (PR #49's workflow comment says "2.16 covers OTP 24-26", which is + wrong for OTP 26 but harmless.) +* Upstream ONNX Runtime stopped publishing x86_64 macOS binaries with 1.24, and `ort` rc.11+ has no + Intel macOS downloads, so `x86_64-apple-darwin` artifacts only work while Ortex stays on rc.8. +* The checksum file must be generated for each release before `mix hex.publish` (PR #49 ships an + empty `%{}` placeholder). +* glibc baseline: artifacts built on `ubuntu-22.04` need glibc 2.35+, which excludes Debian 11, + Ubuntu 20.04, RHEL 8 and Amazon Linux 2. Build Linux artifacts in an older container (manylinux) + or with `cargo-zigbuild` to widen support. After the `ort` upgrade the floor is set by pyke's + static ONNX Runtime instead: glibc 2.39 / libstdc++ 13.2 (Ubuntu 24.04+, Debian 13+), whatever the + build runner. Supporting older distributions then means linking Microsoft's ONNX Runtime (glibc 2.28+) + and shipping it next to the NIF. +* rustler_precompiled facts (0.10.1): default NIF version list is `["2.15"]`, and a 2.15 NIF loads on + OTP 22 to 29, so one artifact per target is enough; `variants` select alternative artifacts per + target at compile time (the mechanism for CUDA builds); extraction unpacks every file in the + tarball into `priv/native`, so side libraries can ship next to the NIF; require `>= 0.10.1` + (earlier versions fail on GitHub's large response headers). +* The static ONNX Runtime still links `libstdc++` dynamically, so minimal runtime images (and Nerves + root filesystems) must include it. + +## 9. Cross-compiling, Nerves, Docker, offline + +* **Cross-compiling / Nerves (issue #41).** The prebuilt table in section 3 has no 32-bit ARM, no + musl and no RISC-V; those targets need an ONNX Runtime built for the target and `ORT_LIB_LOCATION`. + For aarch64 Nerves (rpi4/5, CM4) a community recipe works with Hex `0.1.10` + (`abelino/ortex_fw`, `config/target.exs`): + + ```elixir + config :ortex, Ortex.Native, + target: "aarch64-unknown-linux-gnu", + env: [{"CC", ""}, {"CFLAGS", ""}, + {"CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER", "aarch64-nerves-linux-gnu-gcc"}] + ``` + + Why it is needed: Nerves exports `CC`/`CFLAGS` for the target, `cc`-based build scripts that run on + the host (the `ort-sys` downloader's TLS stack, the unused `rustls`/`ring` dependency) pick them up + and fail, and without `target:` the crate builds for the host. Ortex could derive all of this from + the Nerves environment. The device needs `libstdc++`. +* **Docker (issue #43):** the build image needs Rust, a C/C++ toolchain for linking, CA certificates + (for the download) and network access. After the `ort` upgrade it also needs OpenSSL headers on + Linux (rc.10+ downloads through native TLS) and Rust 1.92+ (rc.14+). The runtime image needs the NIF + and `libstdc++` (static CPU builds), plus `libonnxruntime*` and the CUDA libraries for GPU builds. +* **Offline builds:** download or build ONNX Runtime ahead of time and set `ORT_LIB_LOCATION`. +* **Large models and the dirty IO stack (issue #42):** ONNX Runtime's graph optimisation can recurse + deeply. Session creation runs on a dirty IO scheduler thread (default stack 40 kilowords, 320 KiB + on 64-bit), and a 300 MB model overflowed it and killed the VM. Workarounds today: a lower + optimisation level, or a larger dirty IO stack with `+sssdio ` (e.g. in + `rel/vm.args.eex` or `ERL_FLAGS`). The fix is to build sessions on a Rust thread with a large stack. + +## 10. After the `ort` rc.15 upgrade (planned) + +What changes in this document once Ortex moves to `ort` rc.15 (details in `upgrading.md` section 3): + +| Topic | rc.8 (today) | rc.15 | +| --- | --- | --- | +| Download host and format | `parcel.pyke.io`, `.tgz` | `https://cdn.pyke.io/0/pyke:ort-rs/ms@1.31.0/[+].tar.lzma2`, SHA-256 pinned, GitHub attestations | +| Download cache | `ort-sys` cache directory | `$ORT_CACHE_DIR`, else `~/.cache/ort.pyke.io` (Linux) or `~/Library/Caches/ort.pyke.io` (macOS); cache it in CI | +| Linking | static for CPU, dynamic for CUDA (`cu12`) | always a static core (`libonnxruntime.a`); CUDA/TensorRT packages add provider `.so` files loaded at run time | +| Side libraries next to the NIF | `libonnxruntime*` for CUDA builds | only `libonnxruntime_providers_*` (and WebGPU's Dawn library) | +| Custom ONNX Runtime | `ORT_LIB_LOCATION` | `ORT_LIB_PATH` (old name still read, removed before 2.0.0); `load-dynamic` with `ort::init_from(path)` or an absolute `ORT_DYLIB_PATH` | +| Build host | Rust 1.70+, no OpenSSL | Rust 1.92+; OpenSSL headers on Linux unless the `tls-rustls` feature is used | +| Prebuilt targets | 6 (incl. Intel macOS) | Apple Silicon macOS 13.4+, x86_64/aarch64 Linux (glibc 2.39+, AVX2 on x86_64), x86_64/aarch64 Windows; no Intel macOS, musl, armv7 or riscv | +| Rust 1.100 | dynamic builds fail (section 2) | supported | +| Feature/target validation | none | `ort` reports impossible provider sets as a link error, but `-undefined dynamic_lookup` (macOS) hides it; validate in Elixir and load the NIF in CI | diff --git a/docs/elixir-layer.md b/docs/elixir-layer.md new file mode 100644 index 0000000..03b1595 --- /dev/null +++ b/docs/elixir-layer.md @@ -0,0 +1,159 @@ +# Elixir layer: public API and Nx integration + +Snapshot: `main` at `55dc3de` with the locked dependencies (`nx 0.6.2`, `rustler 0.29.1`). Behaviour +marked "verified" was observed by running the code on Elixir 1.20.4 / OTP 29.1.2 (see +[`testing-and-ci.md`](testing-and-ci.md#4-behaviour-probe-results)). + +## 1. Public API + +### `Ortex.load(path, eps \\ [:cpu], opt \\ 3)` + +Delegates to `Ortex.Model.load/3`, which calls `Ortex.Native.init/3` on a dirty IO scheduler. + +| Argument | Accepted | Behaviour | +| --- | --- | --- | +| `path` | binary string | Charlists and other terms raise `ArgumentError`. A missing file raises `RuntimeError, "File at `...` does not exist"` (verified). | +| `eps` | list of atoms, highest priority first: `:cpu`, `:cuda`, `:tensorrt`, `:acl`, `:onednn`, `:coreml`, `:directml`, `:rocm` | Unknown atoms silently become CPU (verified with `:bogus`). Providers that were not compiled in or are unavailable at run time silently fall back to CPU (verified with `:cuda` on macOS). Strings raise `ArgumentError` (verified). | +| `opt` | integer | `1`, `2`, `3` select ONNX Runtime graph optimisation levels basic, extended, all; any other integer disables graph optimisation. | + +Returns `%Ortex.Model{reference: ref}`. There is no `{:ok, _}`/`{:error, _}` variant and no keyword +options (thread counts, memory settings, config entries, loading from memory: see `TODO.md`). + +### `Ortex.run(model, inputs)` + +Delegates to `Ortex.Model.run/2`. + +* `inputs` is a tuple of tensors in model input order, or a single tensor (wrapped into a 1-tuple). +* Each input is moved to `Ortex.Backend` with `Nx.backend_transfer/2` (a copy if it lives elsewhere). +* Returns a tuple of `Nx.Tensor`s on `Ortex.Backend`, one per model output, in model output order. + A single-output model returns `{tensor}`. +* ONNX `bool` outputs come back as `{:u, 8}`. A `bool` model input needs a `{:u, 8}` tensor holding + only 0 and 1; anything else panics inside the NIF (`ErlangError :nif_panicked`). +* Errors: wrong dtype or shape raise `RuntimeError` with ONNX Runtime's message (verified, the + messages are good). Too few inputs raise an unhelpful kernel error from deep inside the graph + (verified). **Extra inputs are silently ignored** (verified). + +### `Ortex.Model` + +`%Ortex.Model{reference: ref}` where `ref` is the NIF resource. The `Inspect` implementation calls +`Ortex.Native.show_session/1`. Actual output (verified): + +```text +#Ortex.Model< + inputs: [{"x", "Tensor {\n ty: Int32,\n dimensions: [\n -1,\n 100,\n ],\n}", [-1, 100]}, ...] + outputs: [...]> +``` + +The README and the module doc still show the older format (`{"x", "Int32", [nil, 100]}`): the type +string is now `ort`'s multi-line `Debug` output and dynamic dimensions are `-1`, not `nil`. + +### `Ortex.Serving` + +Implements `Nx.Serving`. `Nx.Serving.new(Ortex.Serving, model)` works inline (`Nx.Serving.run/2`) and +as a process (`{Nx.Serving, serving: ..., name: ..., batch_size: ..., batch_timeout: ...}` plus +`Nx.Serving.batched_run/2`); both verified with `models/tinymodel.onnx`. + +* `init(_type, model, [_defn_options])` only accepts **exactly one** partition. Nx passes one keyword + list per partition (each with `:batch_keys`); with `partitions: true` that is `max_concurrency` + lists for `Nx.Defn.Evaluator` and one per device for EXLA, so `init/3` raises + `FunctionClauseError` (verified with Nx 1.0). +* `handle_batch/3` materialises the batch with `Nx.Defn.jit_apply(&Function.identity/1, [batch])` + (the code calls it "a hack"; it is in fact what Nx's own `Nx.Batch` docs use), so stacking runs on + the default `defn` compiler. It then calls `Ortex.run/2` synchronously before returning + `{:execute, fn -> {out, :server_info} end, state}`. +* That ordering is a problem. Nx.Serving calls `handle_batch/3` inside its GenServer and runs the + returned function in a supervised task, so a failing batch only fails that batch. Ortex runs the + inference before returning the function, so inference blocks the serving process and an + exception kills the whole serving. Materialisation and `Ortex.run/2` belong inside the `:execute` + function. `Nx.LazyContainer.traverse/3` can materialise a batch without `defn`. +* `jit` and `defn` pre/post-processing inside the serving are not supported (module doc). + +## 2. `Ortex.Backend` + +A storage-only `Nx.Backend`: struct `%Ortex.Backend{ref: resource}`. The module also derives +`Nx.Container` for its struct (`containers: [:ref]`), which nothing uses; a reference is not a +container, so traversing it would fail. Remove it unless a test proves it is needed. + +At compile time the module computes `Nx.Backend.behaviour_info(:callbacks) -- Module.definitions_in(__MODULE__, :def)` +and defines every missing callback to raise `"operation is not yet supported on Ortex.Backend."`. +`Ortex.Backend.__unimplemented__/0` returns that list. + +### Callback coverage (Nx 0.6.2: 132 callbacks) + +| Status | Callbacks | +| --- | --- | +| Implemented (9) | `init/1`, `from_binary/3`, `to_binary/2`, `backend_transfer/3`, `inspect/2`, `slice/5`, `reshape/2`, `squeeze/3`, `concatenate/3` | +| Raise (123) | everything else, including the data-movement ones a storage backend should have: **`backend_copy/3`**, **`backend_deallocate/1`**, `as_type/2`, `to_batched/3`, `bitcast/2`, `constant/3`, `iota/3`, `eye/2`, `transpose/3`, `broadcast/4`, `pad/4`, `put_slice/4`, `reverse/3`, `select/4` | +| Optional callbacks also stubbed | `optional/3`, `top_k/3`, `svd/3`, `solve/3`, `phase/2`, `logical_not/2`, `determinant/2`, `cumulative_*/3`, `all_close/4`. Defining them stops Nx from using its built-in fallbacks; the generator should skip `behaviour_info(:optional_callbacks)`. | + +### What changes with Nx 1.0.0 (115 callbacks, none optional) + +Because the stubs are generated from `behaviour_info`, Ortex compiles and its tests pass on Nx 1.0.0 +unchanged (verified). The callback set did change: + +| Change | Callbacks | Effect on Ortex | +| --- | --- | --- | +| Added | `stack/3` (Nx 0.8), `to_pointer/2` and `from_pointer/5` (0.8), `block/4` (0.12, replaces `optional/3`) | **`Nx.stack/2` on Ortex tensors now raises.** Before Nx 0.8 the front end built `stack` from `new_axis` + `concatenate`, which Ortex supports. Implement `stack/3` with reshape + the concatenate NIF. Pointers are not needed. | +| Removed (21) | `all_close/4`, `cholesky/2`, `cumulative_*/3`, `determinant/2`, `eigh/3`, `logical_not/2`, `lu/3`, `map/4`, `optional/3`, `phase/2`, `qr/3`, `random_normal/4`, `random_uniform/4`, `solve/3`, `svd/3`, `take/4`, `take_along_axis/4`, `top_k/3` | None (stubs disappear). The "optional callbacks are stubbed" problem goes away. | +| Arity changed | `gather/4`, `indexed_add/5`, `indexed_put/5` | None (stubs regenerate). | + +Other Nx 1.0 facts that matter here: + +* The default integer type is `{:s, 32}` since Nx 0.9 (`Nx.tensor([1])`), so examples that print + `s64` are out of date. +* `%Nx.Tensor{}` gained `donatable?: false`; `Ortex.Model.run/2` builds structs by hand and gets the + default, which is fine. +* With mixed backends, Nx 1.0 hands the operation to the non-`BinaryBackend` backend, so + `Nx.concatenate([ortex_tensor, binary_tensor])` reaches `Ortex.Backend.concatenate/3` and fails + with `MatchError` (verified). It should transfer and cast inputs instead. +* `backend_deallocate/1` must return `:ok | :already_deallocated`. `Nx.BinaryBackend` implements + `backend_copy/3` by delegating to `backend_transfer/3`. Nx.Serving's distributed docs recommend + `Nx.backend_copy(&1, Nx.BinaryBackend)`, which raises on Ortex outputs today. +* `to_binary(tensor, limit)` must return exactly the first `limit` elements (Nx passes + `min(size, limit)`); for sub-byte types the result may be a bitstring. +* Slice start indices: all-integer starts arrive pre-clamped; if any start is a tensor, Nx turns + every start into a scalar tensor and the backend must clamp to `[0, dim - len]` itself. +* New numeric types exist: `{:s, 2}`, `{:s, 4}`, `{:u, 2}`, `{:u, 4}`, `{:f, 8}` (E5M2, Nx 0.9) and + `{:f8_e4m3fn, 8}` (Nx 0.11). All of them panic the NIF today (verified). +* `Nx.Shape.squeeze/3`, which `Ortex.Backend.squeeze/3` calls, is a private API (`@moduledoc false`); + Nx already passes the squeezed shape and names in `out`. + +Implementation notes and known deviations from the Nx contract: + +| Callback | Deviation | +| --- | --- | +| `to_binary/2` | Ignores `limit`; always returns the whole tensor (verified: `Nx.to_binary(t, limit: 2)` on a 6-element `f32` tensor returned 24 bytes, expected 8). | +| `backend_transfer/3` | Has a dead clause for `Nx.Tensor` as a target backend. The generic clause returns a zero-copy resource binary to the target backend (fine). Does not deallocate the source (fine for a GC-managed resource). | +| `backend_copy/3` | Not implemented, so `Nx.backend_copy(ortex_tensor, Nx.BinaryBackend)` raises (verified). | +| `backend_deallocate/1` | Not implemented, so `Nx.backend_deallocate/1` raises (verified). Should return `:ok`. | +| `slice/5` | Only integer start indices. Tensor start indices (allowed by Nx, which then clamps them) raise `ArgumentError` (verified). Calls a non-dirty NIF that copies data. | +| `reshape/2`, `squeeze/3`, `concatenate/3`, `slice/5` | Do not check for `{:error, msg}` from the NIF. An error would be stored as the tensor's `ref`. | +| `concatenate/3` | Raises unless all inputs have the same type (Nx would normally have cast them already). | +| `inspect/2` | Uses `min(:infinity, size)`, which works only because atoms sort after numbers. Adds an `Ortex.Backend` line when `config :ortex, add_backend_on_inspect: true` (the default; this repo turns it off in `:test`). | + +## 3. Data types + +| Nx type | Ortex (`OrtexTensor`) | ONNX type | Status | +| --- | --- | --- | --- | +| `{:s, 8}` `{:s, 16}` `{:s, 32}` `{:s, 64}` | `s8`..`s64` | `INT8`..`INT64` | supported in and out | +| `{:u, 8}` `{:u, 16}` `{:u, 32}` `{:u, 64}` | `u8`..`u64` | `UINT8`..`UINT64` | supported in and out | +| `{:f, 16}` `{:bf, 16}` `{:f, 32}` `{:f, 64}` | `f16`, `bf16`, `f32`, `f64` | `FLOAT16`, `BFLOAT16`, `FLOAT`, `DOUBLE` | supported in and out | +| `{:u, 8}` holding 0/1 | internal `bool` | `BOOL` | inputs converted when the model wants bool; outputs returned as `{:u, 8}` | +| `{:c, 64}` `{:c, 128}` | none | `COMPLEX64`, `COMPLEX128` | **panics** on transfer (verified); `ort` rc.15's `num-complex` feature could back them | +| (no Nx type) | none | `STRING` | output hits `todo!()` (panic) | +| `{:s, 4}` `{:u, 4}` `{:s, 2}` `{:u, 2}` (Nx 0.9+) | none | `INT4`, `UINT4`, `INT2`, `UINT2` | unsupported; panics on transfer (verified) | +| `{:f, 8}` (E5M2), `{:f8_e4m3fn, 8}` (Nx 0.9+, 0.11+) | none | `FLOAT8E5M2`, `FLOAT8E4M3FN` | unsupported; panics on transfer (verified) | +| (no Nx type) | none | `FLOAT8E4M3FNUZ`, `FLOAT8E5M2FNUZ` | unsupported | +| (none) | none | `FLOAT4E2M1`, `FLOAT8E8M0`, `FLOAT6*` | unsupported | +| (containers) | none | sequence, map, optional values | output **panics** | + +## 4. Configuration keys + +| Key | When read | Effect | +| --- | --- | --- | +| `config :ortex, add_backend_on_inspect: boolean` | compile time (`Application.compile_env`) | Adds the `Ortex.Backend` line to tensor `inspect` output. | +| `config :ortex, Ortex.Native, features: ["cuda", ...]` | compile time | Cargo features passed to the crate (selects which ONNX Runtime build `ort` downloads and which providers can register). Must be set in the **consuming application's** config; this repo's `config/config.exs` is ignored for dependents. | +| Other `config :ortex, Ortex.Native, ...` keys | compile time | Passed through to Rustler (e.g. `mode`, `target`, `env`), with the caveats of whichever Rustler version resolves. | + +`config/config.exs` also sets `config :elixir, :dbg_callback, {Macro, :dbg, []}` with a comment that +"something is setting this to IEx.Pry". That is leftover debugging configuration; remove it. diff --git a/docs/execution-providers.md b/docs/execution-providers.md new file mode 100644 index 0000000..02c8394 --- /dev/null +++ b/docs/execution-providers.md @@ -0,0 +1,123 @@ +# Execution providers + +An execution provider (EP) is the ONNX Runtime backend that runs (part of) a graph: CPU, CUDA, +TensorRT, CoreML, DirectML and so on. A session gets an ordered list of providers; ONNX Runtime +assigns each graph node to the first provider that supports it and falls back to CPU. + +Snapshot: `main` at `55dc3de` (`ort 2.0.0-rc.8`, ONNX Runtime 1.19.2). Changes in newer `ort` and +ONNX Runtime releases are summarised in section 5 and detailed in [`upgrading.md`](upgrading.md). + +## 1. How Ortex selects providers today + +`Ortex.load(path, eps)` passes a list of atoms. `utils::map_eps` turns each into +`ort::ExecutionProvider::default().build()` and the session builder registers them in order. + +| Atom | `ort` type (rc.8) | Cargo feature | Prebuilt ONNX Runtime (rc.8 download) | Notes | +| --- | --- | --- | --- | --- | +| `:cpu` | `CPUExecutionProvider` | none | all six targets | Always available. | +| `:cuda` | `CUDAExecutionProvider` | `cuda` | `cu12` variant on x86_64 Linux and Windows | Needs CUDA 12 + cuDNN at run time. Dynamic library (see build doc). | +| `:tensorrt` | `TensorRTExecutionProvider` | `tensorrt` | same `cu12` variant | Also needs TensorRT libraries installed. | +| `:coreml` | `CoreMLExecutionProvider` | `coreml` | macOS builds include it | The feature is required for registration even though the binary is the same. | +| `:directml` | `DirectMLExecutionProvider` | `directml` | Windows | Feature required for registration. | +| `:acl` | `ACLExecutionProvider` | (none forwarded) | no | Arm Compute Library; needs a custom ONNX Runtime. | +| `:onednn` | `OneDNNExecutionProvider` | (none forwarded) | no | Needs a custom ONNX Runtime. (`:dnnl` exists as an atom but is not mapped.) | +| `:rocm` | `ROCmExecutionProvider` | (none forwarded) | no | Removed from ONNX Runtime in 1.23. | +| anything else | `CPUExecutionProvider` | | | **Silent fallback.** | + +`ort` only registers a provider when its Cargo feature is on; otherwise registration returns an error. +`ort` treats registration errors as non-fatal by default and only logs them with `tracing::warn!`. +Ortex installs no `tracing` subscriber, so the warning is lost: **asking for an unavailable provider +silently runs on CPU** (verified: `[:cuda]` and `[:bogus]` on macOS). `ort` offers +`ExecutionProviderDispatch::error_on_failure()` to make registration failures fatal; Ortex does not use it. + +## 2. Enabling a provider in an application + +1. Choose features in the application's config (the Ortex repo's own `config/config.exs` is not used + for dependents): + + ```elixir + config :ortex, Ortex.Native, features: ["cuda", "tensorrt"] + ``` + +2. Recompile Ortex (`mix deps.compile ortex --force`). With `cuda`/`tensorrt` on x86_64 Linux or + Windows, `ort` downloads the dynamic CUDA 12 build of ONNX Runtime. +3. Make the ONNX Runtime side libraries loadable next to the NIF (`priv/native`). On `main` this + relies on `Ortex.Util.copy_ort_libs/0`, which does not find them with Rustler >= 0.37 (the + version new installs resolve). GPU builds also fail to compile on Rust 1.100 until Ortex moves + to `ort` rc.14+ (see `build-and-packaging.md`). +4. Install the provider's system libraries (CUDA 12 runtime, cuDNN, TensorRT) on the machine that runs + the code. +5. Load with the provider first and CPU last: `Ortex.load(path, [:cuda, :cpu])`. + +There is no API to ask which providers are compiled in or available, and no way to pass provider +options (device id, memory limit, TensorRT engine cache, CoreML flags...). An unmerged maintainer +branch, `tm-execution-providers` (March 2024), sketches option structs such as +`%Ortex.CUDAExecutionProvider{device_id: ..., gpu_mem_limit: ...}` and +`%Ortex.TensorRTExecutionProvider{...}`; it is useful prior art for the API. + +## 3. Interaction with BEAM scheduling + +Inference runs on a dirty IO scheduler while ONNX Runtime uses its own intra-op thread pool (by +default one per session, sized to the machine, with busy-wait spinning enabled). Several loaded +models, or a model next to busy BEAM schedulers, oversubscribe the CPU. Useful settings once Ortex +exposes session options (PRs #50, #51, #52): `intra_op_num_threads`, `inter_op_num_threads`, +`session.intra_op.allow_spinning = "0"`, and a shared global thread pool. + +## 4. GPU memory and lifetimes + +Tensors cross the boundary through host memory: inputs are copied into ONNX Runtime-owned tensors +and outputs are copied out into `ndarray`s. Nothing keeps data on the GPU between calls (no +IoBinding). A model's GPU memory is released when the `%Ortex.Model{}` is garbage collected. + +## 5. What changes with newer `ort` and ONNX Runtime + +Upstream ONNX Runtime: + +* **1.23** removed the ROCm provider (AMD now points to MIGraphX or Vitis AI) and introduced the + plugin execution-provider API. +* **1.24** added CUDA 13 support and stopped publishing x86_64 macOS binaries (minimum macOS 14). + CUDA 12 packages are deprecated (still published through 1.31). +* **1.28 to 1.31** also release CUDA and WebGPU as separate *plugin* providers that must match the core + version (1.31.0 pairs with CUDA plugin 0.3.0). The in-tree CUDA provider is still the default in the + core GPU packages. 1.31 brings CoreML into device-based provider selection and deprecates ACL; DirectML + is in "sustained engineering". Microsoft's official binaries from 1.29 can send telemetry + (`ORT_DISABLE_TELEMETRY=1` turns it off). + +`ort` (details and sources in [`upgrading.md`](upgrading.md) section 3.3): + +* **rc.11** renamed `XXXExecutionProvider` to `ort::ep::XXX` and builds pyke binaries with thread + spinning off by default (re-enable it for servers). +* **rc.12** made `AutoDevicePolicy::MaxEfficiency` the session default; registering `ep::CPU` does not + override it, so with ONNX Runtime 1.31 a `[:cpu]` session on macOS can run on CoreML (pykeio/ort#662). + Use `with_auto_device(AutoDevicePolicy::PreferCPU)` for CPU-only sessions. +* **rc.13** compiles provider structs only with their Cargo feature, reports unsatisfiable feature sets + as an unresolved-symbol link error (`lax-feature-matching` relaxes it), and ships CUDA 13 only + (CUDA >= 13.2 and cuDNN >= 9.23; Turing or newer GPUs). +* **rc.14** adds CUDA builds for Blackwell (x64) and aarch64 (Jetson Orin/Thor, Grace) and MIGraphX on + x64 Linux. + +Prebuilt provider packages in rc.15: + +| Target | Packages | +| --- | --- | +| `aarch64-apple-darwin` | `coreml`; `coreml,webgpu` | +| `x86_64-unknown-linux-gnu` | CPU; `openvino`; `webgpu`; `nvrtx`; `migraphx`; `cuda13,tensorrt,nvrtx` | +| `aarch64-unknown-linux-gnu` | CPU; `cuda13,tensorrt,nvrtx` | +| `x86_64-pc-windows-msvc` | `directml`; `webgpu`; `openvino`; `nvrtx,directml`; `cuda13,tensorrt,nvrtx,directml` | +| `aarch64-pc-windows-msvc` | `directml` | + +Providers without prebuilt packages (ROCm, QNN, XNNPACK, ACL, oneDNN, CANN, Vitis, ...) need a custom +ONNX Runtime through `ORT_LIB_PATH`. In CUDA packages the core is static but the provider libraries +(`libonnxruntime_providers_{shared,cuda,tensorrt,nv_tensorrt_rtx}.so`) are loaded at run time and must +sit next to the NIF. + +Consequences for Ortex: + +* Drop `:rocm` (or map it to MIGraphX) and consider `:migraphx`, `:openvino`, `:webgpu`, `:nvrtx`. +* Gate each provider with `#[cfg(feature = ...)]` and return a clear error when an unavailable one is + requested. +* `-undefined dynamic_lookup` on macOS hides `ort`'s link-time check: a bad feature set builds a NIF + without ONNX Runtime that only fails at load. Validate feature/target combinations in Elixir and load + the NIF in CI. +* Make "available providers" queryable (rc.14 has `EP::is_available()`), and decide whether GPU support + ships as a precompiled `rustler_precompiled` variant or stays source-build only. diff --git a/docs/native-layer.md b/docs/native-layer.md new file mode 100644 index 0000000..ae1824f --- /dev/null +++ b/docs/native-layer.md @@ -0,0 +1,153 @@ +# Native layer (Rust NIF crate) + +Snapshot: `main` at `55dc3de`. Crate: `native/ortex` (`crate-type = ["cdylib"]`, `edition = "2018"`). +How the crate is compiled and linked is in [`build-and-packaging.md`](build-and-packaging.md). + +## 1. Dependencies (as pinned on `main`) + +| Crate | Pin (Cargo.toml) | Locked | Used for | Notes | +| --- | --- | --- | --- | --- | +| `rustler` | `0.29.0` | `0.29.1` | NIF macros, resources, term codecs | `rustler::resource!` triggers 2 `non_local_definitions` warnings on Rust 1.99 | +| `ort` | `2.0.0-rc.8` | `2.0.0-rc.8` | ONNX Runtime bindings | default features: `ndarray`, `half`, `download-binaries`, `copy-dylibs`; bundles ONNX Runtime 1.19.2 | +| `ndarray` | `0.16.1` | `0.16.1` | owned n-d arrays inside `OrtexTensor` | must be the same major version `ort` uses, or `ort`'s ndarray conversions will not accept the arrays | +| `half` | `2.2.1` | `2.3.1` | `f16`, `bf16` element types | | +| `tracing-subscriber` | `0.3` | | nothing (only a commented-out call in `model.rs`) | unused, remove | +| `num-traits` | `0.2.15` | | nothing | unused, remove | +| `rustls` | `0.22.4` | | nothing directly | pinned to force a patched transitive version (dependabot alert on `ort-sys`'s downloader); remove when `ort` is upgraded | + +Cargo features forwarded to `ort`: `directml`, `coreml`, `cuda`, `tensorrt`. Which ones are enabled +comes from `config :ortex, Ortex.Native, features: [...]` (see build doc). + +## 2. NIF table + +All NIFs are registered in `rustler::init!("Elixir.Ortex.Native", [...], load = ...)` in `lib.rs` and +stubbed in `lib/ortex/native.ex`. + +| Elixir stub | Rust fn | Arguments (decoded as) | Returns | Scheduler | Data movement | +| --- | --- | --- | --- | --- | --- | +| `init(path, eps, opt)` | `lib.rs::init` | `String`, `Vec`, `i32` | `ResourceArc` or `{:error, String}` | DirtyIo | reads model file | +| `run(model, inputs)` | `lib.rs::run` | `ResourceArc`, `Vec>` | `[{tensor_ref, [dims], dtype_atom, bits}]` or `{:error, String}` | DirtyIo | clones every input, copies every output | +| `from_binary(bin, shape, type)` | `lib.rs::from_binary` | `Binary`, shape tuple, `{atom, bits}` | `ResourceArc` | DirtyCpu | copies the binary (unsafe, see section 5) | +| `to_binary(ref, bits, limit)` | `lib.rs::to_binary` | `ResourceArc`, `usize`, `usize` | resource binary aliasing the tensor | DirtyCpu | zero-copy; `bits` and `limit` are **ignored** | +| `show_session(model)` | `lib.rs::show_session` | `ResourceArc` | `{[{name, debug_string, dims}], [...]}` | normal | metadata only | +| `slice(ref, starts, lengths, strides)` | `lib.rs::slice` | `Vec` x3 | `ResourceArc` | normal (should be dirty) | copies the slice | +| `reshape(ref, shape)` | `lib.rs::reshape` | `Vec` | `ResourceArc` or `{:error, String}` | normal (should be dirty) | clones the whole array | +| `concatenate(refs, type, axis)` | `lib.rs::concatenate` | `Vec`, `{atom, bits}`, `i32` | `ResourceArc` | normal (should be dirty) | copies twice (concatenate, then re-layout) | + +Error convention: `rustler::Error::Term(Box::new(string))` encodes as `{:error, string}` (a return +value, not an exception). Decoding failures return `badarg`, which Elixir sees as `ArgumentError`. + +## 3. Types + +### `OrtexModel` (`model.rs`) + +```rust +pub struct OrtexModel { pub session: ort::Session } +unsafe impl Sync for OrtexModel {} +``` + +Registered as a resource in `lib.rs`'s `load` callback (`rustler::resource!(OrtexModel, env)`). +Shared by reference across BEAM processes; `run` uses `&Session` concurrently (valid for +`ort 2.0.0-rc.8` where `Session::run(&self, ...)`). + +### `OrtexTensor` (`tensor.rs`) + +An enum with one variant per element type, each holding an owned `ndarray::Array`: + +| Variant | Rust element | Nx type | ONNX element type | +| --- | --- | --- | --- | +| `s8`, `s16`, `s32`, `s64` | `i8`..`i64` | `{:s, 8..64}` | `INT8`..`INT64` | +| `u8`, `u16`, `u32`, `u64` | `u8`..`u64` | `{:u, 8..64}` | `UINT8`..`UINT64` | +| `f16` | `half::f16` | `{:f, 16}` | `FLOAT16` | +| `bf16` | `half::bf16` | `{:bf, 16}` | `BFLOAT16` | +| `f32`, `f64` | `f32`, `f64` | `{:f, 32}`, `{:f, 64}` | `FLOAT`, `DOUBLE` | +| `bool` | `bool` | (none; internal only) | `BOOL` | + +Methods: `shape`, `reshape`, `dtype` (returns `(atom, bits)`), `to_bytes`, `slice`, `to_bool`; free +function `concatenate` (macro-expanded per dtype). `Clone` is implemented by hand. + +Conversions: + +* `TryFrom<&ort::Value> for OrtexTensor`: matches `ValueType::Tensor { ty, .. }`, extracts with + `try_extract_tensor::()?.into_owned()` (a copy), maps `Bool` to `u8`, `todo!()` on `String`, + `panic!` on non-tensor values. +* `TryFrom<&OrtexTensor> for ort::SessionInputValue`: `arr.clone().try_into::()` (a copy). + +## 4. Other helpers (`utils.rs`, `constants.rs`) + +* `from_binary`: matches the `(type_string, bits)` pair and builds the variant from an unsafe view + over the BEAM binary followed by `.to_owned()`. +* `to_binary`: `reference.make_binary(env, |t| t.to_bytes())`. +* `map_eps`: atom string to `ort::*ExecutionProvider::default().build()` for `cpu`, `cuda`, + `tensorrt`, `acl`, `onednn`, `coreml`, `directml`, `rocm`; **anything else silently maps to CPU**. + The `dnnl` atom exists in `constants.rs` but is not mapped. +* `map_opt_level`: `1|2|3` map to `Level1|2|3`; any other integer disables graph optimisation. +* `is_bool_input`: true when the model input's element type (or a map/sequence/optional's inner + type) is `Bool`. + +## 5. Unsafe code inventory + +| Location | What | Risk | Fix | +| --- | --- | --- | --- | +| `model.rs` `unsafe impl Sync for OrtexModel` | Declares the session shareable across threads | Redundant with rc.8 (the session is already shareable); becomes **unsound** with `ort` versions where `run` needs `&mut self` | Remove; on newer `ort`, wrap state in a lock or a session pool (see `upgrading.md`) | +| `utils.rs` `initialize_from_raw_ptr` | `ArrayViewMut::from_shape_ptr(shape, bin.as_ptr() as *mut T)` over BEAM binary memory | (1) no check that `bin.len() == product(shape) * size_of::()`: a short binary reads out of bounds; (2) BEAM binaries (especially sub-binaries) are not guaranteed to be aligned for `T`; (3) creates a *mutable* view of shared immutable memory | Validate the length, copy bytes into a `Vec` (`copy_nonoverlapping` or `bytemuck::pod_collect_to_vec`), build with `Array::from_shape_vec`. PR #47 does this. | +| `tensor.rs` `get_bytes` | `std::mem::zeroed()` of a generic element just to learn its size; `slice::from_raw_parts(array.as_ptr(), len * size)` | Correct only for standard-layout arrays (invariant), needless `zeroed` | Use `size_of::()` and `array.as_slice_memory_order()` (returns `None` if not contiguous) or `bytemuck::cast_slice` | + +Every `unsafe` block should carry a `// SAFETY:` comment stating the invariant it relies on. + +## 6. Panic inventory + +Rustler turns a panic into an `ErlangError` with reason `:nif_panicked`; nothing crashes the VM, +but the caller gets no useful message. + +| Location | Trigger | Reachable from Elixir? | +| --- | --- | --- | +| `utils.rs::from_binary` `unimplemented!()` | Nx type without a variant: `{:c, 64}`, `{:c, 128}`, and the newer Nx sub-byte and 8-bit float types | **Yes** (verified with `{:c, 64}`) | +| `tensor.rs::to_bool` `panic!` | A model bool input receives a non-`u8` tensor (e.g. `Nx.tensor([1, 0])`, which is `s64`) or a `u8` value other than 0/1 | **Yes** | +| `tensor.rs` `TryFrom<&Value>` `panic!` | Model output is a sequence, map or optional (e.g. scikit-learn classifiers with ZipMap) | **Yes** | +| `tensor.rs` `TryFrom<&Value>` `todo!` | Model output is a string tensor | **Yes** | +| `tensor.rs` `shape/reshape/dtype/to_bytes/slice` `panic!` | Called on the internal `bool` variant | No (bool tensors never reach Elixir) | +| `tensor.rs` `concatenate!` `.unwrap()` | Shapes that cannot be concatenated | Not via Nx (Nx validates shapes first) | +| `tensor.rs::concatenate` `unimplemented!()` | Unsupported dtype | Not via Nx | +| `model.rs::run` `.expect(..)` | Output name missing from `SessionOutputs` | Not normally | +| `tensor.rs::slice_array` index / `slice_each_axis` bounds | Start/length vectors that do not match the rank or exceed a dimension | Not via Nx with integer indices | + +## 7. `ort` API surface used (rc.8) + +This is the list to re-check on every `ort` upgrade. The rc.15 replacements (checked in the rc.15 +source and by a compile probe) are below; [`upgrading.md`](upgrading.md) explains the changes by +release candidate. + +| rc.8 (today) | rc.15 | +| --- | --- | +| `use ort::{Session, SessionInputValue}` | `ort::session::{Session, SessionInputValue, SessionOutputs, RunOptions}` | +| `ort::GraphOptimizationLevel::Level3` | `ort::session::builder::GraphOptimizationLevel::All` (`Level3` now means "layout") | +| `ort::{Value, DynValue, ValueType, TensorElementType}` | `ort::value::{Value, DynValue, ValueType, TensorElementType, Tensor, TensorRef, Outlet}` | +| `ort::CUDAExecutionProvider::default().build()` | `ort::ep::CUDA::default().build()`, only with the `cuda` feature (same pattern for every provider) | +| `ort::ExecutionProviderDispatch` | `ort::ep::ExecutionProviderDispatch` | +| `Session::builder()?` | `Session::builder(&env)?` with the process-wide `Environment` from `ort::init().build()?` | +| `session.inputs` / `session.outputs` fields; `.input_type` | `session.inputs()` / `session.outputs()` returning `&[Outlet]`; `outlet.name()`, `outlet.dtype()` | +| `ValueType::tensor_dimensions() -> Option<&Vec>` | `ValueType::tensor_shape() -> Option<&Shape>` (`Shape` derefs to `[i64]`) | +| `ValueType::Tensor { ty, dimensions }` | `ValueType::Tensor { ty, shape, dimension_symbols }` | +| `session.run(&inputs)` with `&self` | `session.run(&inputs)` with `&mut self` (lock or pool) | +| `value.try_extract_tensor::()?.into_owned()` | `value.try_extract_array::()?.to_owned()` | +| `array.clone().try_into::()` | `Tensor::from_array(owned)?` or zero-copy `TensorRef::from_array_view(view)?` | +| exhaustive `match` on `TensorElementType` | wildcard arm required (`#[non_exhaustive]`; new Complex, Float8, Int4/Uint4, Int2/Uint2, Float4, Float8E8M0, Undefined variants) | +| `f16`/`bf16` extraction works by default | requires `features = ["half"]` | + +* Crate-root imports: `ort::{Session, Error, ExecutionProviderDispatch, GraphOptimizationLevel, Value, DynValue, SessionInputValue, TensorElementType, ValueType}`. +* Session building: `Session::builder()?`, `.with_optimization_level(..)?`, `.with_execution_providers(Vec)?`, `.commit_from_file(path)?`. No environment is created explicitly. +* Execution providers: `ort::{CPU,CUDA,TensorRT,ACL,OneDNN,CoreML,DirectML,ROCm}ExecutionProvider::default().build()`. +* Introspection: `session.inputs` / `session.outputs` (public fields), `.name`, `.input_type` / `.output_type`, `ValueType::tensor_dimensions()`, `Debug` formatting of `ValueType`, matching `ValueType::{Tensor, Map, Sequence, Optional}`. +* Inference: `session.run(&[SessionInputValue])`, `SessionOutputs::get(name)`, `Value::dtype()`, `Value::try_extract_tensor::()`, `Array: TryInto`. +* Element types matched exhaustively: `TensorElementType::{Bfloat16, Float16, Float32, Float64, Uint8, Uint16, Uint32, Uint64, Int8, Int16, Int32, Int64, String, Bool}`. + +## 8. Rust warnings on the current toolchain + +Building `main` with Rust 1.99.1 produces 15 warnings and no errors: + +* 12x `mismatched_lifetime_syntaxes` from the `concatenate!` macro's `filter` helper (`tensor.rs`). +* 1x the same lint on `initialize_from_raw_ptr` (`utils.rs`). +* 2x `non_local_definitions` from `rustler::resource!` in `lib.rs` (fixed by Rustler 0.34+ `#[rustler::resource_impl]`). + +There is no `cargo fmt` or `cargo clippy` step in CI. diff --git a/docs/testing-and-ci.md b/docs/testing-and-ci.md new file mode 100644 index 0000000..61d1c7c --- /dev/null +++ b/docs/testing-and-ci.md @@ -0,0 +1,139 @@ +# Testing and CI + +## 1. Running the tests + +```sh +mix deps.get +MIX_ENV=test mix compile --warnings-as-errors # also builds the Rust crate and downloads ONNX Runtime +mix format --check-formatted +mix test # resnet50 tests are excluded unless models/resnet50.onnx exists +mix test --only resnet50 # after generating the model, see below +``` + +* The first build needs network access: `ort`'s build script downloads ONNX Runtime. +* **Editing Rust code does not trigger a rebuild.** `lib/ortex/native.ex` builds the crate by hand + and registers no external resources, so `mix compile` skips it (verified by touching + `native/ortex/src/utils.rs`). Use `mix compile --force` or `touch lib/ortex/native.ex`. +* Use `MIX_ENV=test` when you only need the library and its tests: the `:dev`-only deps (`exla`, + `torchx`, `tokenizers`) download large native artifacts. +* Rust-only checks: `cd native/ortex && cargo fmt --check && cargo clippy`. Running cargo directly + builds into `native/ortex/target` (ignored by git) and downloads ONNX Runtime again. + +## 2. Test suite layout + +| File | Covers | Notes | +| --- | --- | --- | +| `test/test_helper.exs` | Excludes `:resnet50` tests unless `models/resnet50.onnx` exists | | +| `test/ortex_test.exs` | `doctest Ortex`; resnet50 inference and `Nx.Serving` (tagged); `Nx.Serving` with `tinymodel` (batch of 3, two inputs of different dtypes, three outputs) | The `Ortex.load/3` doctests only check that loading does not raise. | +| `test/dtype/dtype_test.exs` | Binary round trip BinaryBackend → Ortex → BinaryBackend for all 12 supported dtypes plus a scalar | | +| `test/shape/concat_test.exs` | `Nx.concatenate` on Ortex tensors (1-D, 3-D, axes 1 and 2, three inputs, mixed types raise) | **Bug:** the type table maps every name except `s8` to `{:s, 16}`, so 10 of the 12 dtype variants are never exercised. PR #47 fixes the table. | +| `test/shape/reshape_test.exs` | `Nx.reshape` including `:auto` and a tensor-as-shape | Has a `TODO` saying it does not truly validate the Ortex side. | +| `test/shape/slice_test.exs` | `Nx.slice` 1-D and 2-D, with and without strides, `f32` and `u8` | Integer start indices only. | +| `test/shape/squeeze_test.exs` | `Nx.squeeze` with and without axes, with names | | + +### Test models + +| Model | Where | How to (re)create | +| --- | --- | --- | +| `models/tinymodel.onnx` | committed (8.7 KB) | `python python/multi_input.py` (PyTorch, opset 19). The script writes `tinymodel.onnx` into the **current directory**; move it to `models/`. Inputs: `x` int32 `[batch, 100]`, `y` float32 `[batch, 100]`; three float32 `[batch, 10]` outputs. | +| `models/resnet50.onnx` | not committed (~100 MB) | `python python/export_resnet.py` (torchvision, opset 19). Also writes into the current directory. The script uses the deprecated `pretrained=True` torchvision argument; PR #47 modernises it. | + +## 3. Verified results (2026-10-29) + +`main` (`55dc3de`) was cloned into a scratch directory and built on macOS arm64 with +Elixir 1.20.4, Erlang/OTP 29.1.2 (NIF 2.18) and Rust 1.99.1, `MIX_ENV=test`: + +| Step | Result | +| --- | --- | +| `mix deps.get` | OK (locked: nx 0.6.2, rustler 0.29.1, complex 0.5.0, telemetry 1.2.1, jason 1.4.1, toml 0.7.0) | +| `mix compile --warnings-as-errors` | OK in about 16 s. Hundreds of warnings, **all from dependencies** (nx 0.6.2: pin-operator and charlist deprecations; jason, toml, rustler 0.29.1: charlists, `~~~`, typing). None from Ortex's own Elixir code. | +| Rust crate | OK, 15 warnings (see `native-layer.md` section 8). ONNX Runtime 1.19.2 is linked statically into a 21 MB `libortex.so` (with CoreML). No `libonnxruntime` side library is produced on macOS. | +| `mix format --check-formatted` | OK | +| `mix test` | **117 passed** (2 doctests, 115 tests), 2 excluded (`:resnet50`) | + +Conclusion: the current code already runs on the newest Elixir/OTP; what blocks real-world use is +dependency constraints and drift (for example `{:nx, "~> 0.6"}` cannot resolve together with Nx 1.0). +See [`upgrading.md`](upgrading.md). + +Further runs on the same machine: + +| Run | Result | +| --- | --- | +| Fresh consumer app depending on `main` (dependency resolution without the lock) | Resolves Nx 0.13.1 and Hex Rustler 0.38.0; compiles; `Ortex.run` on `tinymodel.onnx` works. Rustler 0.38 builds in `native/ortex/target/release` and installs `priv/native/ortex.so`. | +| `mix.exs` bumped to `nx ~> 1.0`, `rustler ~> 0.38`, `ex_doc ~> 0.40`, `elixir ~> 1.17` (crate unchanged) | compile with `--warnings-as-errors`, `format`, `mix docs` OK; 117 tests pass | +| Rust crate bumped to `rustler 0.39` | Three layers of compile errors (see `upgrading.md`); after mechanical fixes, 117 tests pass on OTP 29 | +| Rust **1.100.0-beta.7**, static CPU build | OK (new cargo lint: `rustls` and `tracing-subscriber` are unused dependencies) | +| Rust **1.100.0-beta.7**, dynamically linked ONNX Runtime (`ORT_LIB_LOCATION`, `copy-dylibs` path) | **Fails**: `ort-sys` rc.8 build script panics (`build.rs:111`, "No such file or directory") because the new build-dir layout moves `OUT_DIR`. Same build passes on 1.99.1. Affects CUDA/TensorRT builds, including this repo's Linux CI. | +| `mix test --warnings-as-errors` | Fails after a fully passing run because `test/test_helper.exs` uses `IO.warn` when `models/resnet50.onnx` is missing. | + +## 4. Behaviour probe results + +A throwaway script exercised edge cases against the same build. These are the facts the docs +and `TODO.md` rely on; each should become a regression test. + +| Case | Observed | +| --- | --- | +| `inspect(model)` | Type strings are multi-line `ort` `Debug` output; dynamic dims are `-1` (docs say `nil`). | +| `Ortex.run` with too few inputs | `RuntimeError` from a Gemm kernel ("Non-zero status code returned while running Gemm node ...") | +| `Ortex.run` with an extra input | Succeeds; the extra input is silently ignored. | +| Wrong dtype / wrong shape | `RuntimeError` with clear ONNX Runtime messages. | +| `Ortex.load` missing file | `RuntimeError, "File at `./models/nope.onnx` does not exist"` | +| `Ortex.load(path, [:bogus])` / `[:cuda]` on macOS | Loads on CPU, no warning. | +| `Ortex.load(path, ["cpu"])` | `ArgumentError, "argument error"` | +| `Nx.backend_copy(t, Nx.BinaryBackend)` | `RuntimeError, "operation backend_copy is not yet supported on Ortex.Backend."` | +| `Nx.backend_deallocate(t)` | `RuntimeError, "operation backend_deallocate is not yet supported ..."` | +| `Nx.to_binary(t, limit: 2)` on 6 x f32 | 24 bytes (limit ignored) | +| `Nx.tensor([1.0], type: :c64) \|> Nx.backend_transfer(Ortex.Backend)` | `ErlangError :nif_panicked` (`not implemented` at `utils.rs:78`) | +| `Nx.slice(t, [Nx.tensor(1), 0], [1, 3])` | `ArgumentError, "argument error"` | +| 32 concurrent tasks x 20 `Ortex.run` on one model | OK (runs in parallel on rc.8) | +| Process-based `Nx.Serving.batched_run` | OK | +| `Nx.Serving` with `partitions: true` and 2 partitions (Nx 1.0) | `FunctionClauseError` in `Ortex.Serving.init/3` | +| `Nx.stack/2` on Ortex tensors (Nx 1.0; any Nx >= 0.8) | `RuntimeError, "operation stack is not yet supported on Ortex.Backend."` | +| `Nx.concatenate([ortex_tensor, binary_tensor])` (Nx 1.0) | `MatchError` in `Ortex.Backend.concatenate/3` | +| `{:s, 4}`, `{:u, 2}`, `{:f, 8}`, `{:f8_e4m3fn, 8}` tensors to Ortex (Nx 1.0) | `ErlangError :nif_panicked` | + +Memory probe (issue #44): a GenServer kept the latest outputs of `tinymodel.onnx` in its state while +running 600 inferences with 20,000-row batches (about 8 MB of input and 2.4 MB of output per run). + +| Variant | RSS | +| --- | --- | +| Fixed batch size | 139 MB at start, about 188 MB after 100 runs, flat until 600 runs, 143 MB after `:erlang.garbage_collect/1` on the holder | +| Random batch size (1 to 20,000) | 109 MB at start, peak 287 MB after 100 runs, drifting down to 201 MB at 600 runs, 183 MB after GC | + +The BEAM's own memory stayed near 75 MB throughout. No unbounded growth was reproduced on macOS with +rc.8: native tensor memory waits for garbage collection, and ONNX Runtime's arena keeps some memory. + +## 5. Gaps in the test suite + +* No tests for error paths (bad path, wrong input count, wrong dtype/shape, unknown provider). +* No test model with a `bool` input or output (bool support from PR #39 is untested), none with + string, sequence or map outputs, none with multiple partitions in `Nx.Serving`. +* No concurrency, memory-growth (issue #44) or large-tensor tests. +* No tests for `inspect(model)` format, `backend_copy`, `backend_deallocate`, `to_binary` limits. +* No Rust unit tests (`cargo test`), no `cargo clippy`/`cargo fmt` in CI. +* resnet50 tests never run in CI (no model), so real-model inference is untested in CI. + +## 6. CI (`.github/workflows/ci.yml`) + +Two jobs on push to `main` and on pull requests, both `MIX_ENV=test`, steps +`deps.get` → `compile --warnings-as-errors` → `format --check-formatted` → `test`: + +| Job | Runner | Toolchain | +| --- | --- | --- | +| `linux` | `ubuntu-latest` | Elixir 1.17.3 / OTP 27.1.2 via `erlef/setup-beam@v1` | +| `macos` | `macos-latest` | Homebrew `erlang@27` and whatever `elixir` Homebrew ships | + +Gaps and problems: + +* `actions/checkout@v2` is long deprecated (runs on an old Node runtime); use the current major. +* Single version pair; nothing tests the declared minimum (`elixir: "~> 1.14"`) or the newest + Elixir/OTP. The macOS job's Elixir version floats with Homebrew. +* No caching of `deps`, `_build`, `~/.cargo` or the ONNX Runtime download. +* On Linux the repo's `config/config.exs` enables the `cuda` and `tensorrt` features, so CI + downloads the CUDA build of ONNX Runtime and runs the tests on its CPU provider. That is a + different binary from what a dependent gets by default (CPU-only, statically linked). It also + makes Linux CI depend on pyke's CDN for a 111 MB download instead of the 18 MB CPU package (a + transient 404 failed PR #50) and + will break when the runners move to Rust 1.100 (see the table in section 3). +* No Windows job, no Rust lint job, no docs build (`mix docs`), no Hex package build check. +* No release workflow; releases are cut by hand. PR #49 adds a precompiled-NIF release workflow. diff --git a/docs/upgrading.md b/docs/upgrading.md new file mode 100644 index 0000000..efc0bd2 --- /dev/null +++ b/docs/upgrading.md @@ -0,0 +1,378 @@ +# Upgrading Ortex: Elixir ecosystem, `ort` and ONNX Runtime + +Snapshot: 2026-10-29, baseline `main` at `55dc3de`. "Verified" means the step was run in a scratch +copy on macOS arm64 (Elixir 1.20.4, OTP 29.1.2, Rust 1.99.1, and Rust 1.100.0-beta.7 where noted). +Facts about `ort` come from its release notes and from its source at each git tag. Task IDs +(COMPAT-01, ORT-03, ...) refer to [`../TODO.md`](../TODO.md). + +## 0. Where things stand + +| Component | `main` | Target | Why it matters | +| --- | --- | --- | --- | +| Elixir | `~> 1.14` | `~> 1.17` | Nx 1.0's floor; Elixir 1.20 is needed for OTP 29 | +| Nx | `~> 0.6` (lock 0.6.2) | `~> 0.6 or ~> 1.0` for 0.1.x, `~> 1.0` for 0.2 | `~> 0.6` excludes 1.0, so Ortex cannot sit next to Nx/EXLA/Bumblebee 1.0 | +| Rustler (Hex) | `~> 0.27` (lock 0.29.1; new installs get 0.38.0) | `~> 0.38`, `runtime: false` | private-API use in `native.ex`; build layout changed in 0.37 | +| Rustler (crate) | `0.29` | `0.39` | `resource!` and `init!` lists removed; warnings on current Rust | +| ex_doc | `0.29.4` (exact) | `~> 0.40` | | +| `ort` | `2.0.0-rc.8` | `=2.0.0-rc.15` | ONNX Runtime 1.19.2 to 1.31.0, Rust 1.100 compatibility, 30+ UB fixes | +| ONNX Runtime | 1.19.2 (opset 21, IR 10) | 1.31.0 (opset 26) | newer exported models, Blackwell GPUs, security fixes | +| `ndarray` | `0.16` | `0.17` | must match `ort`'s | +| Rust (MSRV) | rc.8 needs 1.70 | 1.92 (rc.14+) | CI and Docker images | + +`ort` versions and the ONNX Runtime each downloads (from the `ort-sys` distribution list at each tag): + +| `ort` | Released | ONNX Runtime | API features (default) | Binary host | +| --- | --- | --- | --- | --- | +| rc.8 | 2024 | 1.19.2 | (single version) | `parcel.pyke.io` (still serving) | +| rc.9 | 2024-11-21 | 1.20.0 | | `parcel.pyke.io` | +| rc.10 | 2025-06-05 | 1.22.0 | | `cdn.pyke.io` | +| rc.11 | 2026-01-07 | 1.23.2 | | `cdn.pyke.io` | +| rc.12 | 2026-03-05 | 1.24.2 | `api-17`..`api-24` (`api-24`) | `cdn.pyke.io` | +| rc.13 | 2026-07-28 | 1.28.0 | `api-17`..`api-28` (`api-27`) | `cdn.pyke.io` | +| rc.14 | 2026-10-12 | 1.31.0 | `api-17`..`api-31` (`api-31`) | `cdn.pyke.io` | +| rc.15 | 2026-10-19 | 1.31.0 | `api-17`..`api-31` (`api-31`) | `cdn.pyke.io` | + +No date is announced for `ort` 2.0.0 ("No definite timeline", pykeio/ort#451). Do not wait for it. + +## 1. Recommended order + +1. **Patch release `0.1.11` on `ort` rc.8.** Widen Nx, fix the crashes and the GPU library copy, keep CI + alive on Rust 1.100. Small, low-risk, unblocks most users. +2. **Modernise without touching `ort`** (the "modernisation" half of PR #47): Rustler crate and + package upgrade, plain `use Rustler`, the Nx 1.0 contract fixes and the serving fixes. +3. **`ort` rc.15** (the "ort" half of PR #47, retargeted from rc.11). +4. **Session options**: one keyword API (consolidating PRs #50, #51, #52). +5. **Precompiled NIFs** (PR #49) against the final NIF and ONNX Runtime. + +Each step builds and passes the full test suite on its own; dependent steps can be stacked PRs. + +## 2. Elixir ecosystem (all verified) + +### 2.1 `mix.exs` + +For `0.1.11` keep the floor low and widen Nx: + +```elixir +{:nx, "~> 0.6 or ~> 1.0"}, +``` + +For the modernisation release: + +```elixir +def project do + [elixir: "~> 1.17", ...] +end + +defp deps do + [ + {:rustler, "~> 0.38", runtime: false}, + {:nx, "~> 1.0"}, + {:ex_doc, "~> 0.40", only: :dev, runtime: false} + # exla/torchx are unused; tokenizers only serves examples: move examples to Mix.install + ] +end +``` + +Then `mix deps.unlock --unused` to drop stale lock entries (axon, axon_onnx, ...). With these changes +and no Elixir code changes, `mix compile --warnings-as-errors`, `mix format --check-formatted`, +`mix docs` and all 117 tests pass on Elixir 1.20.4 / OTP 29.1.2 (verified). + +### 2.2 Rust crate: Rustler 0.29 to 0.39 + +`cargo check` with `rustler = "0.39"` fails in three layers; fix each to reveal the next (verified): + +1. `rustler::init!` no longer accepts a function list (NIFs are discovered automatically; the list was + ignored from 0.34 and removed in 0.38), and `rustler::resource` is now private: + + ```rust + use rustler::ResourceArc; // was rustler::resource::ResourceArc + rustler::init!("Elixir.Ortex.Native"); // no list, no load callback needed for resources + ``` + +2. `rustler::resource!` is gone (deprecated 0.34, removed 0.38) and `ResourceArc` now requires + `T: Resource`: + + ```rust + #[rustler::resource_impl] + impl rustler::Resource for OrtexModel {} + + #[rustler::resource_impl] + impl rustler::Resource for OrtexTensor {} + ``` + +3. NIF return types must be `RefUnwindSafe` (since 0.35) and `ort::Session` is not: + + ```rust + impl std::panic::RefUnwindSafe for OrtexModel {} + ``` + +After that only lints remain. Other facts: MSRV 1.91; the default NIF version is still 2.15 (loads on +OTP 22 to 29; `nif_version_2_18` exists since 0.38); `Error::Term` still returns `{:error, term}`; +panics are still caught and raised as `:nif_panicked`; 0.39 requires `panic = "unwind"`. + +### 2.3 `lib/ortex/native.ex` + +Replace the version switch around the private `Rustler.Compiler.compile_crate` and the +`skip_compilation?: true` with: + +```elixir +defmodule Ortex.Native do + @moduledoc false + use Rustler, otp_app: :ortex, crate: :ortex + + # Still needed only while some builds ship side libraries (e.g. CUDA provider .so files). + Ortex.Util.copy_ort_libs() + + # NIF stubs ... +end +``` + +Verified: tests pass, and editing Rust sources now triggers a rebuild (Rustler registers the crate +files as external resources only when it compiles the crate itself). + +Rustler 0.37+ builds in `native/ortex/target//` and writes `priv/native/ortex.so` into the +source tree, so add `/priv/native/` to `.gitignore`, and make `copy_ort_libs/0` read the real target +directory (`cargo metadata --format-version=1 --no-deps` gives `target_directory`; add `/` when +`:target` is set and the profile from `:mode`). + +### 2.4 `.cargo/config.toml` + +With crate 0.36+, macOS needs only the rpath flag: + +```toml +[target.'cfg(target_os = "macos")'] +rustflags = ["-C", "link-args=-Wl,-rpath,@loader_path"] +``` + +Verified with crate 0.39 and the static ONNX Runtime 1.19.2 (all tests pass). Keep the Linux `$ORIGIN` +rpath entries. Dropping `-undefined dynamic_lookup` has a second benefit after the `ort` upgrade: +`ort`'s "unsatisfiable execution providers" error is a deliberate unresolved symbol, and the flag +hides it (section 3.3). + +### 2.5 Nx contract work exposed by newer Nx + +| Item | What to do | TODO | +| --- | --- | --- | +| `Nx.stack/2` raises (Nx >= 0.8 made it a callback) | implement `stack/3`: reshape each input with a new axis, then the concatenate NIF | BUG-14 | +| `backend_copy/3`, `backend_deallocate/1` raise | delegate copy to the transfer logic; return `:ok` | BUG-05 | +| `to_binary/2` ignores `limit` | return exactly `limit` elements (`binary_part/3` on the resource binary is zero-copy) | BUG-04 | +| tensor start indices in `slice/5` | `Nx.to_number/1` each, clamp to `[0, dim - len]` | BUG-06 | +| mixed backends or types in `concatenate/3` | transfer non-Ortex inputs and cast instead of `MatchError`/`RuntimeError` | BUG-15 | +| new types (`{:s, 4}`, `{:u, 2}`, `{:f, 8}`, `{:f8_e4m3fn, 8}`, complex) | raise `ArgumentError` in Elixir before the NIF; never `unimplemented!()` | SAFE-02 | +| `squeeze/3` calls private `Nx.Shape.squeeze/3` | use `out.shape` and `out.names` | BUG-12 | + +### 2.6 `Ortex.Serving` + +* Accept any list of partition options in `init/3` (BUG-08). +* Do the batch materialisation and `Ortex.run/2` inside the function returned from `handle_batch/3`, + so inference does not block the serving process and a failing batch does not kill it (BUG-16). +* Prefer `Nx.Defn.Evaluator` (or `Nx.LazyContainer.traverse/3`) for materialising the batch so the + application's default compiler (often EXLA) is not involved (PERF-06). + +### 2.7 CI + +Test the floor and the newest versions: Elixir 1.17 / OTP 25 to 27 and Elixir 1.20.4 / OTP 29.1, plus +Nx 0.x and 1.0 while both are allowed. Use current `actions/checkout`, `erlef/setup-beam` on every +OS, and make Linux CI CPU-only (the CUDA build will not compile on Rust 1.100 with rc.8). + +## 3. `ort` rc.8 to rc.15 (ONNX Runtime 1.19.2 to 1.31.0) + +### 3.1 What the upgrade buys and costs + +Buys: + +* ONNX Runtime 1.31.0: opset 26 instead of 21, newer contrib ops, Blackwell GPU support (issue #45), + aarch64 CUDA builds for Jetson Orin/Thor and Grace (issue #41), MIGraphX builds for AMD on x64 Linux. +* Compatibility with Rust 1.100's build-dir layout (rc.14), more than 30 undefined-behaviour fixes + and several memory-leak fixes (rc.15). +* Every pyke package is a single **static** `libonnxruntime.a`, so a CPU NIF is self-contained + (verified: a 27.4 MB release NIF on macOS arm64 that depends only on system libraries). +* APIs for strings, sequences and maps, complex numbers (`num-complex` feature), output selection, + run cancellation and loading from memory (section 4). + +Costs (platform floor of pyke's prebuilt binaries): + +| Platform | Requirement | +| --- | --- | +| Linux (x86_64, aarch64) | **glibc >= 2.39 and libstdc++ >= 13.2** (Ubuntu 24.04+, Debian 13+), per `ort`'s docs and confirmed by symbol inspection (`__isoc23_strto*`, `std::ios_base_library_init`). rc.8's libraries needed far less (around glibc 2.32), so Ubuntu 22.04, Debian 12 and RHEL 9 lose prebuilt support. | +| x86_64 | AVX2 (x86-64-v3 baseline since rc.11) | +| macOS | arm64 only, macOS >= 13.4 (no Intel macOS binaries since rc.11) | +| musl, armv7/arm, i686, riscv, FreeBSD | no prebuilt binaries | +| CUDA | CUDA >= 13.2 and cuDNN >= 9.23 (no CUDA 12 since rc.13); Turing (sm_75) or newer, so Pascal and Volta are out | +| Build host | Rust >= 1.92; OpenSSL headers on Linux (downloads use `tls-native`; the `tls-rustls` feature avoids OpenSSL) | + +For older Linux hosts, link Microsoft's official ONNX Runtime 1.31.1 (glibc >= 2.28) through +`ORT_LIB_PATH` with `ORT_PREFER_DYNAMIC_LINK=1` and ship it next to the NIF, or use `load-dynamic`. + +### 3.2 Breaking changes by release candidate + +| RC | Change | Ortex code affected | +| --- | --- | --- | +| rc.9 | Crate-root re-exports removed ("undo the flattening"); only `ort::{Error, ErrorCode, Result, init, init_from}` stay at the root. `Value::dtype()` returns `&ValueType`; `ValueType::Tensor` gains `dimension_symbols`; `SessionOutputs` no longer derefs to a `BTreeMap` (`.get(name)` still works). | every `use ort::...`; the `ValueType` pattern in `tensor.rs` | +| rc.10 | `Session::run(&mut self)`. `try_extract_tensor` renamed to `try_extract_array` (returns a view; the old name now returns `(&Shape, &[T])`). All `TryFrom` impls for values removed: use `Tensor::from_array(owned)` or zero-copy `TensorRef::from_array_view` (standard layout). `ValueType::Tensor { ty, shape, dimension_symbols }`; `tensor_dimensions()` became `tensor_shape()`. **`half` is no longer a default feature.** New element types (Complex, Float8, Int4/Uint4, Undefined). Binaries move to `cdn.pyke.io`, become static, and download with native TLS. MSRV 1.81. | model sharing; 13 extraction calls and 13 input conversions in `tensor.rs`; `model::show`; `Cargo.toml` | +| rc.11 | `session.inputs`/`outputs` fields became methods returning `&[Outlet]` (`name()`, `dtype()`). `execution_providers` renamed `ep`, `XXXExecutionProvider` renamed `XXX` (old names removed in rc.13/rc.14). `ndarray 0.17`. **`GraphOptimizationLevel::Level3` now means `ORT_ENABLE_LAYOUT`; the new `All` means `ORT_ENABLE_ALL`.** Binaries: no Intel macOS, AVX2 baseline, Clang builds, and a client-package build that **disables thread spinning by default** (the notes advise re-enabling inter/intra-op spinning for servers). MSRV 1.88. | `model.rs` introspection; `utils.rs` providers and optimisation level | +| rc.12 | Multiversioning with `api-*` features. **`ORT_LIB_LOCATION` renamed `ORT_LIB_PATH`** (rc.15 still reads both; the old name will not reach 2.0.0). `ort::tensor` merged into `ort::value` (breaks PR #47's `ort::tensor::TensorElementType`). Builder methods return `BuilderResult` (`?` still works). **New default `AutoDevicePolicy::MaxEfficiency`** (see 3.3). | docs, `util.ex`, imports | +| rc.13 | Provider structs exist only with their Cargo feature (`#[cfg]` each arm). Unsatisfiable feature sets are reported through a deliberate link error (`lax-feature-matching` relaxes it). `TensorElementType` is `#[non_exhaustive]`. No CUDA 12 builds. Sessions with map inputs/outputs no longer panic on commit. | `utils.rs::map_eps`, `constants.rs`, `Cargo.toml` features | +| rc.14 | **Manual `Environment`:** `let env = ort::init().build()?; Session::builder(&env)?`. **Only one environment per OS process, ever** (verified: a second `build()` fails with "only one environment is allowed per process" even after dropping the first). `ort::execution_providers` removed; `EP::NAME`/`is_available()`. Rust 1.100 build-dir support; Blackwell x64, aarch64 CUDA and MIGraphX binaries; MSRV 1.92. | `model::init`; a new environment owner | +| rc.15 | 30+ UB fixes and memory-leak fixes. `run_async` takes owned inputs; `ExecutionProviderLibrary::unregister` removed; `with_external_initializer_file_in_memory` takes `Arc<[u8]>`. | none directly | + +### 3.3 Design decisions to make first + +**Environment (ORT-04).** There can be exactly one ONNX Runtime environment per OS process, and its +options (name, telemetry, logger, global thread pool, default providers) are fixed when it is first +built. Every `Session` holds a clone of it (it is an `Arc`), so it lives as long as any model. +Therefore: + +* Build it once, inside the NIF: in the Rustler `load` callback (options passed from application + config through `use Rustler, load_data: ...`) or lazily on the first `Ortex.load`. Keep it in a + `static OnceLock` (lazy fallible initialisation needs a `Mutex>` or + `once_cell`). A verified run with a static environment, sessions leaked as BEAM resources are at + `halt`, and a normal exit was clean three times out of three on macOS arm64. +* `ort`'s request that libraries "allow the user to pass their own `Environment`" becomes: let the + application configure the one environment (e.g. `config :ortex, :environment, global_thread_pool: + [...], telemetry: false, log_level: :warning`). +* Set `with_telemetry(false)` (it defaults to true; pyke's binaries contain no telemetry client, but + Microsoft's 1.29+ binaries used with `load-dynamic` do; also set `ORT_DISABLE_TELEMETRY=1` there). +* A global thread pool (`with_global_thread_pool`) replaces one intra-op pool per model. Sessions then + use it unless built with `with_independent_thread_pool()`, so per-session thread options must raise + or force an independent pool in that case. +* Logging: `with_logger(LoggerFunction)` receives every ONNX Runtime log line. To reach Elixir's + `Logger`, push lines into a channel drained by one dedicated OS thread that sends them to a + registered process with `OwnedEnv::send_and_clear` (the callback runs on ONNX Runtime threads). +* Do not purge or reload the NIF module while ONNX Runtime threads may be running. + +**Concurrency (ORT-03, PERF-04, SAFE-06).** `run`, `run_with_options`, `run_binding` and `run_async` +all take `&mut self`. `ort`'s documentation says this is because some ONNX Runtime internals (certain +provider allocators, statistics) are not thread safe and "previous versions of `ort` that allowed +concurrent inference often saw crashes and memory corruption"; its maintainer has declined to revert +it (pykeio/ort#415, #597) and to add a session pool (#469). Upstream, CUDA graphs and DirectML forbid +concurrent `Run()` on one session. Options: + +| Option | Behaviour | Cost | +| --- | --- | --- | +| `Mutex` (PRs #46, #47) | correct; one inference at a time per model | throughput regression for concurrent callers; blocked callers occupy dirty IO threads (10 by default), so bound concurrency on the Elixir side (batching, a queue) | +| Pool of N sessions per model | parallel up to N | memory per session; CPU sessions can share weights with `SessionBuilder::with_prepacked_weights` | +| One session per `Nx.Serving` partition | matches Nx's partition model | only helps serving users | + +Start with the mutex (simple and safe), cache metadata so `show_session` never locks, add a +`:pool_size` option later, and benchmark concurrent serving before and after. `SessionOutputs` +borrows the session, so collect output names before `run` and copy outputs out before unlocking. +`RunOptions::terminate()` can cancel a running inference from another thread. + +**Execution providers (ORT-05).** + +* One Rust function owns the atom-to-provider table: `ort::ep::{CPU, CUDA, TensorRT, DirectML, + CoreML, MIGraphX, OpenVINO, WebGPU, ...}::default().build()`, each arm behind `#[cfg(feature = ...)]`. + A requested provider that is not compiled in returns an error naming the missing Cargo feature. +* **Auto device policy (pykeio/ort#662, open).** Since rc.12 sessions default to + `AutoDevicePolicy::MaxEfficiency`, and registering `ep::CPU` does not override it, so with ONNX + Runtime 1.31 a `[:cpu]` session on macOS can run on CoreML. When only `:cpu` is requested, call + `.with_auto_device(AutoDevicePolicy::PreferCPU)`. +* **The "unsatisfiable providers" link error does not protect Ortex today.** `ort` reports it as an + unresolved symbol; with `-undefined dynamic_lookup` on macOS the NIF links anyway (verified: a + 740 KB cdylib without ONNX Runtime) and only fails when the BEAM loads it ("symbol not found + '_OrtGetApiBase'"). Linux shared objects allow undefined symbols too. Drop the macOS flag (2.4), + validate feature/target combinations in `native.ex`, and load the NIF in CI on every target. +* Valid prebuilt combinations in rc.15 (`ort-sys` `dist.tsv`): + + | Target | Packages | + | --- | --- | + | `aarch64-apple-darwin` | `coreml`; `coreml,webgpu` | + | `x86_64-unknown-linux-gnu` | CPU; `openvino`; `webgpu`; `nvrtx`; `migraphx`; `cuda13,tensorrt,nvrtx` | + | `aarch64-unknown-linux-gnu` | CPU; `cuda13,tensorrt,nvrtx` | + | `x86_64-pc-windows-msvc` | `directml`; `webgpu`; `openvino`; `nvrtx,directml`; `cuda13,tensorrt,nvrtx,directml` | + | `aarch64-pc-windows-msvc` | `directml` | + + No prebuilt binaries exist for ROCm, QNN, XNNPACK, ACL, oneDNN, CANN, Vitis and others; those need + `ORT_LIB_PATH`. ROCm was removed from ONNX Runtime in 1.23 (use MIGraphX); ACL is deprecated (1.31). +* CUDA packages are a static core plus provider libraries loaded at run time + (`libonnxruntime_providers_shared.so`, `..._cuda.so` (121 MB), `..._tensorrt.so`, + `..._nv_tensorrt_rtx.so`). They must sit next to the NIF (`priv/native`); this is the only remaining + job for `copy_ort_libs/0`. `ort::ep::cuda::preload_dylibs` (feature `preload-dylibs`) can preload the + CUDA 13 runtime libraries. +* Thread spinning is off by default in pyke's builds since rc.11. For servers, expose + `with_intra_op_spinning`/`with_inter_op_spinning` (or the global pool's `with_spin_control`). + +**Optimisation level (ORT-06).** Map Ortex's default `3` to `GraphOptimizationLevel::All` +(`ORT_ENABLE_ALL`), not `Level3` (`ORT_ENABLE_LAYOUT` since rc.11), or expose the five levels as atoms. + +**Metadata (ORT-07, BUG-01).** Read inputs and outputs once in `init` (`session.inputs()` / +`session.outputs()` return `&[Outlet]`; `ValueType::tensor_shape()` gives the shape), convert to plain +Rust data (name, element type, dimensions with `None` for dynamic axes, symbolic names) and store it +next to the session. `show_session` then never locks. `ValueType`'s `Display` (e.g. +`Tensor(dyn, 3, 224, 224)`) is a better human-readable form than `Debug`. + +### 3.4 Measured effort (compile probe, Rust 1.99.1) + +Starting from `main` with only `ort = "=2.0.0-rc.15"` and `ndarray = "0.17"` changed, `cargo check` +reports errors in four waves (each fix reveals the next): + +| Wave | Errors | Causes | +| --- | --- | --- | +| 1 | 41 | name resolution: crate-root imports (`model.rs` 6, `tensor.rs` 18, `utils.rs` 17) | +| 2 | 43 | 13x `Value: TryFrom>` missing; 13x `.into_owned()` on the new `try_extract_tensor` tuple; 7x provider structs gated by features; 4x private `inputs`/`outputs`; 2x `half` types (feature off); `Session::builder()` needs `&Environment`; `&ValueType`; the `dimensions` pattern | +| 3 | 2 | `session.run` needs `&mut`; non-exhaustive `TensorElementType` | +| 4 | 1 | `SessionOutputs` borrow (collect output names first) | + +About 172 changed lines reach a clean build, also clean with `--features coreml` and +`--features cuda,tensorrt`. (The probe built a new environment per load, which would fail on the second +load at run time; see 3.3.) From PR #47's rc.11 code the remaining delta is small: about four import +fixes (`ort::ep::*`, `ort::value::TensorElementType`), `#[cfg]` gating for providers, +`Session::builder(&env)` with the environment lifecycle, plus the semantic fixes it lacks (level `3` to +`All`, `PreferCPU`, `ORT_LIB_PATH`). + +Both `main` and PR #47 use caret requirements (`"2.0.0-rc.8"`, `"2.0.0-rc.11"`), which allow later +release candidates; a plain `cargo update` would jump to rc.15 and break the build. Pin with `=`. +Published Hex packages ship `Cargo.lock`, so released versions are not affected. + +### 3.5 Code migration checklist + +1. `native/ortex/Cargo.toml`: `ort = { version = "=2.0.0-rc.15", features = ["half"] }` (add + `"num-complex"` for complex support, `"tls-rustls"` to avoid OpenSSL), `ndarray = "0.17"`; forward + features `cuda`, `tensorrt`, `coreml`, `directml` (optionally `migraphx`, `openvino`, `webgpu`, + `nvrtx`); drop `rocm`; remove `rustls`, `tracing-subscriber`, `num-traits`; `rust-version = "1.92"`. +2. New `runtime.rs`: owns the single `Environment` and its configuration; hands out session builders. +3. `model.rs`: `Session::builder(env)?`, optimisation level `All`, providers, `PreferCPU` for CPU-only, + `commit_from_file`; metadata snapshot; session behind a mutex (or pool); no `unsafe impl Sync`. +4. `tensor.rs`: inputs through `TensorRef::from_array_view` (zero copy; arrays are standard layout) or + `Tensor::from_array(...)`; outputs through `try_extract_array::()`; a wildcard arm returning an + error; bool stays `u8`. +5. `utils.rs` (or a new `ep.rs`): safe `from_binary` (length check and copy), the provider table with + `cfg` gates and clear errors. +6. `lib.rs`: Rustler 0.39 style (section 2.2) if not done already. +7. Elixir: `config/config.exs` feature sets only for valid combinations; `util.ex` copies only + `libonnxruntime_providers_*` (and WebGPU's Dawn library) and reads `ORT_LIB_PATH` then + `ORT_LIB_LOCATION`; update `inspect` format and docs (platform floor, CUDA 13, opset 26, + serialised runs per model). +8. CI: Rust >= 1.92, cache the `ort` download cache (`ORT_CACHE_DIR`, default `~/.cache/ort.pyke.io` on + Linux and `~/Library/Caches/ort.pyke.io` on macOS), load the NIF on every target. +9. Tests: TODO TEST-01 to TEST-04, plus concurrent `run` from many processes, bool inputs, `f16`/`bf16` + outputs, optimisation levels, provider errors, `[:cpu]` staying on CPU on macOS. + +### 3.6 Using an ONNX Runtime newer than `ort` bundles + +rc.15 supports ONNX Runtime API 31, so it can run against 1.31.1: + +* `ORT_LIB_PATH=` (static preferred, or `ORT_PREFER_DYNAMIC_LINK=1`), or +* the `load-dynamic` feature with `ort::init_from(path)` or `ORT_DYLIB_PATH`. Relative paths resolve + against the executable, which for a NIF is `beam.smp`, so pass absolute paths. `load-dynamic` + refuses libraries older than the compiled API level and only logs for newer ones. + +Other build-time variables in rc.15: `ORT_LIB_PROFILE`, `ORT_SKIP_DOWNLOAD`, `ORT_OFFLINE`, +`ORT_CXX_STDLIB`, `ORT_CUDA_VERSION`, `ORT_CACHE_DIR`; run time: `ORT_DYLIB_PATH`, `ORT_LOG`. + +## 4. After the upgrade: things that become possible + +| Capability | `ort` rc.15 API | TODO | +| --- | --- | --- | +| Load from memory (issue #29; also in rc.8) | `SessionBuilder::commit_from_memory(&[u8])` (copies) or `commit_from_memory_directly` (borrows) | ONNX-08 | +| Cancel a slow load | `SessionBuilder::canceler()` | SAFE-05 | +| String tensors | `try_extract_strings`, `try_extract_string_array`, `Tensor::from_string_array` | ONNX-02 | +| Sequences and maps (scikit-learn ZipMap) | `try_extract_sequence::()`, then `try_extract_map::()` or `try_extract_key_values` | ONNX-03 | +| Complex tensors (`{:c, 64}`, `{:c, 128}`) | `num-complex` feature: `Complex32`, `Complex64` are primitive element types | ONNX-05 | +| Float8, Int4/Uint4, Int2/Uint2, Float4 | no Rust types; build with `DynTensor::new` and copy raw bytes | ONNX-04 | +| Output selection, run tags, cancellation | `Session::run_with_options` with `RunOptions::with_outputs`, `with_tag`, `terminate` | ONNX-06, ONNX-14 | +| External weights from memory | `with_external_initializer_file_in_memory(name, Arc<[u8]>)` | ONNX-09 | +| Device-resident outputs | `Session::create_binding()` + `run_binding` (note open bug #667: `bind_input` can deadlock in rc.15 for borrowed tensors already on the session device) | ONNX-13 | +| Plugin providers and device selection | `Environment::register_ep_library(name, path)`, `env.devices()`, `SessionBuilder::with_devices` | ONNX-12 | +| Profiling, dimension overrides, optimised model output | `with_profiling`, `with_dimension_override`, `with_optimized_model_path` | ONNX-07, ONNX-15 |