Runtime & Framework Selection¶
Decision log from prototype planning
Runtimes vs Frameworks¶
Terminology
The thesis distinguishes between runtimes (low-level inference engines) and frameworks (high-level APIs that wrap runtimes).
| Type | Name | API Level | Underlying Runtime |
|---|---|---|---|
| Runtime | TensorFlow.js | Tensor ops | JS/WASM/WebGL/WebGPU |
| Runtime | ONNX Runtime Web | Session + tensors | WASM/WebGL†/WebGPU/WebNN |
| Runtime | LiteRT.js | Session + tensors | WASM/XNNPACK/WebGPU/WebNN |
| Framework | Transformers.js | Pipeline (one-liner) | ONNX Runtime Web |
| Framework | MediaPipe Tasks | Task-specific API | LiteRT/TFLite (WASM) |
| Framework | ML5.js | Friendly API | TensorFlow.js |
† ONNX Runtime Web WebGL support deprecated (Apr 2024), still functional.
Runtimes to include (3)¶
| Runtime | API Level | Philosophy | Model Format |
|---|---|---|---|
| TensorFlow.js | Tensor ops | "Build it yourself" | TF.js graphs, SavedModel |
| ONNX Runtime Web | Session + tensors | "Manage inference yourself" | ONNX |
| LiteRT.js | Session + tensors | "Google's next-gen runtime" | .tflite |
These three runtimes represent the low-level inference engine approach, where the developer manages tensors, preprocessing, and model loading directly.
LiteRT.js (Jul 2026)
Google's new production WebAI runtime, replacing TF.js as Google's strategic direction. Native C++ compiled to WASM with XNNPACK, WebGPU, and WebNN support. Claims up to 3× faster than TF.js. Can integrate into existing TF.js pipelines via @litertjs/tfjs-interop.
Frameworks to include (3)¶
| Framework | API Level | Underlying Runtime | Philosophy |
|---|---|---|---|
| Transformers.js | Pipeline (one-liner) | ONNX Runtime Web | "Just give me a model" |
| MediaPipe Tasks | Task-specific API | LiteRT/TFLite (WASM) | "Just give me data" |
| ML5.js | Friendly API | TensorFlow.js | "ML for everyone" |
These three frameworks represent the high-level API approach, where the framework handles preprocessing, postprocessing, and model management internally.
Runtimes dropped¶
keras.js — treated as a research dead end. Last commit ~2018, Keras 1.x era. Superseded by TensorFlow.js. Its deprecation and lack of modern backend support is itself a valid thesis finding under the criterion Verbreitung / Community.
Backend support¶
| Backend | TF.js | ORT Web | LiteRT.js | Transformers.js | MediaPipe | ML5.js |
|---|---|---|---|---|---|---|
| WASM | ||||||
| WebGL | ||||||
| WebGPU | ||||||
| WebNN |
ML5.js and MediaPipe backend inheritance
ML5.js inherits all backends from TensorFlow.js (it wraps TF.js directly). MediaPipe Tasks uses WebGL for preprocessing on web and WASM+XNNPACK for inference (its own embedded TFLite runtime, not LiteRT.js). Neither ML5.js nor MediaPipe expose backend selection APIs — backends are managed internally.
ORT Web WebGL deprecation
ONNX Runtime Web deprecated WebGL in Apr 2024 (GitHub #20224). Still functional but WebGPU is recommended. Transformers.js v4 inherits this — WebGL is available via ORT Web but not recommended.
Transformers.js v4 Changes
Transformers.js v4 (Feb 2026) adopted a new C++ WebGPU runtime rewritten from scratch. WebGL support is inherited from ORT Web (deprecated but functional). Primary backends: WASM (default) + WebGPU.
Browser compatibility (caniuse.com, Jul 2026)¶
| Backend | Release Date | Global Usage | Desktop | Mobile | Notes |
|---|---|---|---|---|---|
| WebGL | Jul 2015 | 97.01% — Widely available | All major browsers green | Near-universal | Opera Mini unsupported; requires up-to-date GPU drivers |
| WASM | Oct 2017 | 96.07% — Widely available | All major browsers green | Near-universal | Opera Mini unsupported |
| WebGPU | Apr 2023 | ~86% — Limited availability | Chrome 113+, Edge 113+, Firefox 138+ default (Windows, Apr 2025; more platforms since 141), Safari 26+ (partial) | Chrome Android 150+, Safari iOS 26+, Samsung 24+ | Linux HW-dependent; Safari partial (macOS Tahoe only) |
| WebNN | Apr 2023 | low — Emerging | Chromium-based browsers (Chrome, Edge, derivatives) on supported hardware; validated working on Edge stable/Win11 in benchy | None | Not shipped in Firefox/Safari; availability depends on OS ML stack (DirectML/CoreML) |
Updated July 2026
Earlier drafts described WebGPU on Firefox as flag-gated and WebNN as flag-gated Chrome/Edge-only. Both are outdated: Firefox ships WebGPU by default since 138 (validated in benchy on Firefox/Windows — 2× over Firefox wasm), and WebNN works unflagged in Chromium-based browsers on supported hardware (validated: onnx webnn completed on Edge stable/Win11, truth-reported webnn). WebNN remains excluded from the primary measured matrix for cross-browser comparability since no non-Chromium browser ships it.
Architecture implication
The fallback chain is: WebNN → WebGPU → WebGL → WASM. WebNN is checked first if enabled, then progressively falls back to widely-supported backends. WebGL and WASM provide 96-97% global coverage as safe fallbacks.
Key decisions¶
-
WASM is the CPU execution path — All runtimes that support WASM run on the CPU via WebAssembly. There is no meaningful "pure CPU" backend distinct from WASM. The TF.js
cpubackend (pure JS) is a slower fallback, not a separate execution target. -
Three runtimes, three frameworks — TF.js, ORT Web, and LiteRT.js are low-level inference engines. Transformers.js, MediaPipe Tasks, and ML5.js are high-level wrappers. This gives a symmetric comparison matrix.
-
LiteRT.js is Google's strategic direction — Released Jul 2026, it replaces TF.js as Google's recommended web runtime. Claims 3× faster than TF.js. Runs
.tflitemodels with WebGPU/WebNN acceleration. -
ONNX Runtime Web deprecated WebGL — Apr 2024, still functional. WebGPU is the recommended path. Transformers.js v4 inherits this deprecation.
-
MediaPipe is self-contained on web — Uses its own embedded TFLite WASM runtime, not LiteRT.js. WebGL for preprocessing, WASM+XNNPACK for inference. No WebGPU support yet (open feature request #5826).
-
ML5.js is a thin wrapper — Adds no inference capabilities beyond TF.js. Value is in accessibility (creative coding audience), not performance.
Model distribution plan¶
CDN sources for all model artifacts:
| Source | Used by | Model | Format |
|---|---|---|---|
| TF Hub (via jsDelivr) | TF.js, ML5.js | MobileNet v2 | TF.js graph |
| HuggingFace Hub | Transformers.js | MobileNet v2 | ONNX |
| Google Storage | MediaPipe | EfficientNet-Lite0 | .tflite |
| HuggingFace / TF Hub | LiteRT.js | MobileNet v2 | .tflite |
| HuggingFace Hub | ONNX Runtime Web | MobileNet v2, SqueezeNet 1.1 | ONNX |
ONNX Model Access
ONNX models are loaded directly from HuggingFace's /resolve/main/ CDN path — no authentication required. The ONNX Model Zoo GitHub repo discontinued LFS downloads in July 2025; HuggingFace is the official successor (huggingface.co/onnxmodelzoo).
LiteRT.js Model Format
LiteRT.js uses .tflite models — the same format as MediaPipe. Models can be sourced from HuggingFace, TF Hub, or converted from PyTorch/JAX/TF via the LiteRT converter.
Conversion toolchain
The full conversion path landscape: PyTorch → ONNX via torch.onnx.export; PyTorch → TFLite via litert-torch (official Google, Beta); TF/Keras → TF.js via tensorflowjs_converter; TF/Keras → TFLite via the LiteRT converter; ONNX → TFLite via onnx2tf (community). The PINTO Model Zoo demonstrates inter-conversion across TF, PyTorch, ONNX, OpenVINO, TFJS, TFLite (f32/f16/int8), EdgeTPU, and CoreML. Details: Model & Task Distribution → Model Sources & Conversion.
Future: MODEL_REGISTRY centralizes model URLs and metadata in pipeline.js.