Implementation Plan: Prototype Redesign¶

This document outlines the redesign of the prototype's pipeline system to support maximum user freedom across all configuration layers, re-enable ONNX Runtime, add LiteRT.js, and enable framework overhead measurement.

Terminology¶

Precise naming avoids confusion across the four browser ML runtimes:

Runtime vs Framework¶

A runtime is a self-contained inference engine with its own model format and backend system. A framework is a wrapper that depends on a runtime underneath:

ml5.js is included as a TF.js framework wrapper to enable symmetric overhead measurement across all three runtimes.

Model format is determined by runtime¶

All models load from CDN — no local files in the repository.

Pipeline order¶

The user configures selections in dependency order. Each step constrains what follows:

Task → Model → Runtime → Framework → Backend → Dataset → Run

Cascade rules¶

When a user changes a selection, downstream selections are cleared:

Block system redesign¶

6 layers (up from 5)¶

Framework options by runtime¶

When "Direct" is selected, the adapter calls the runtime API directly. When a framework wrapper is selected, the adapter uses the framework's higher-level API.

Model × Runtime availability¶

The MODEL_REGISTRY maps architecture → runtime → CDN URL. If a model is not available for a runtime, that runtime is greyed out in the UI.

Framework overhead measurement¶

The prototype enables direct comparison: same model, same runtime, with vs without framework wrapper.

This gives 6 benchmark combinations (3 runtimes × 2 modes) for a clean overhead comparison table in the thesis.

Runtime and adapter changes¶

New runtime: LiteRT.js¶

LiteRT.js (released July 9, 2026) is Google's JavaScript binding of the LiteRT C++ runtime via WASM. It loads .tflite models and supports WebGPU, WASM+XNNPACK, and WebNN backends.

Updated adapter list¶

COMPAT matrix update¶

export const COMPAT = {
  ort: {
    webgpu: { supported: true, hw: ["GPU"] },
    wasm:   { supported: true, hw: ["CPU"] },
    webgl:  { supported: false, reason: "ORT dropped WebGL support" },
    cpu:    { supported: false, reason: "Use WASM instead" },
  },
  tfjs: {
    webgl:  { supported: true, hw: ["GPU"] },
    webgpu: { supported: true, hw: ["GPU"] },
    wasm:   { supported: true, hw: ["CPU"] },
    cpu:    { supported: true, hw: ["CPU"] },
  },
  litert: {
    webgpu: { supported: true, hw: ["GPU"] },
    wasm:   { supported: true, hw: ["CPU"] },
    webgl:  { supported: false, reason: "LiteRT.js uses WebGPU, not WebGL" },
    cpu:    { supported: false, reason: "Use WASM instead" },
  },
};

Model registry¶

The MODEL_REGISTRY is the source of truth for which models exist and where to load them from.

const MODEL_REGISTRY = {
  "mobilenet-v2": {
    label: "MobileNet v2",
    description: "Fast, general-purpose image classifier (13 MB)",
    inputSize: 224,
    runtimePaths: {
      ort:    "https://huggingface.co/onnxmodelzoo/mobilenetv2-12/resolve/main/mobilenetv2-12.onnx",
      tfjs:   null,  // loaded via @tensorflow-models/mobilenet CDN package, not a URL
      litert: "https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/5/converted_default/1",
    },
  },
  "squeezenet-1.1": {
    label: "SqueezeNet 1.1",
    description: "Tiny model, fastest load, lower accuracy (~5 MB)",
    inputSize: 224,
    runtimePaths: {
      ort:    "https://huggingface.co/onnxmodelzoo/squeezenet11/resolve/main/squeezenet11.onnx",
      litert: "https://tfhub.dev/google/squeezenet/1.0/classification/1/converted_default/1",
    },
  },
  "efficientnet-lite0": {
    label: "EfficientNet-Lite0",
    description: "Balanced accuracy and speed (~4 MB)",
    inputSize: 224,
    runtimePaths: {
      mediapipe: "https://storage.googleapis.com/mediapipe-models/image_classifier/efficientnet_lite0/float32/1/efficientnet_lite0.tflite",
      litert:    "https://storage.googleapis.com/mediapipe-models/image_classifier/efficientnet_lite0/float32/1/efficientnet_lite0.tflite",
    },
  },
};

Note: The TF.js adapter loads MobileNet via the @tensorflow-models/mobilenet CDN package (not a direct URL), so its runtimePaths entry is null — the adapter handles loading internally.

UI behavior¶

Model layer¶

When the user opens the Model layer, all three models are shown. After selecting a model, incompatible runtimes are greyed out with an explanation:

MobileNet v2    ✅ Click to select
SqueezeNet 1.1  ✅ Click to select
EfficientNet-Lite0  ✅ Click to select

After selecting "MobileNet v2", the Runtime layer shows:

ORT Web       ✅ .onnx (HuggingFace CDN)
TF.js         ✅ model.json (CDN)
LiteRT.js     ✅ .tflite (TF Hub)
MediaPipe     ❌ No model available — use EfficientNet-Lite0

Framework layer¶

After selecting a runtime, the Framework layer shows wrapper options:

[ORT Web selected]
Direct (ORT Web)     — No wrapper overhead
Transformers.js      — Wraps ORT, adds model discovery + preprocessing

[TF.js selected]
Direct (TF.js)       — No wrapper overhead
ml5.js               — Wraps TF.js, friendly API for creative coding

[LiteRT.js selected]
Direct (LiteRT.js)   — No wrapper overhead
MediaPipe Tasks       — Wraps LiteRT, task-specific API

Backend layer¶

After selecting a runtime, backends are filtered by the COMPAT matrix (existing behavior, extended for 3 runtimes).

Files to modify¶

Implementation phases¶

Phase 1: Fix ONNX and restore model selection¶

1. Set ONNX_MODEL_URL to HuggingFace CDN URL
2. Remove the null check and early throw in ONNXAdapter.load()
3. Update COMPAT.onnx → COMPAT.ort with correct supported backends
4. Restore interactive model selection cards in _renderModelCards()
5. Add MODEL_REGISTRY with CDN URLs for all models × runtimes
6. Filter runtime cards based on selected model

Phase 2: Add LiteRT.js runtime¶

1. Implement LiteRTAdapter class (load .tflite from CDN, WASM+XNNPACK backend)
2. Add LiteRT.js to COMPAT matrix
3. Add LiteRT.js to runtime options in LayerRegistry
4. Add CDN URLs for .tflite models to MODEL_REGISTRY

Phase 3: Add framework layer¶

1. Add FRAMEWORK_REGISTRY with per-runtime framework options
2. Implement _renderFrameworkCards() in PipelineCanvas
3. Add framework selection to PipelineState (6^th field)
4. Implement ML5Adapter class (wrap ml5.imageClassifier())
5. Update AdapterFactory to return framework-aware adapters
6. Update cascade rules (runtime change clears framework + backend)

Phase 4: Update tutorial and playground¶

1. Update tutorial.js steps to match new pipeline order
2. Update playground.js to use 6-layer pipeline
3. Test all 6 benchmark combinations (3 runtimes × 2 modes)
4. Update result display to show framework name alongside runtime

Phase 5: Polish and verify¶

1. Verify all CDN URLs work (ORT models from HuggingFace, .tflite from TF Hub/GCS)
2. Test model × runtime filtering (greyed out cards with reasons)
3. Test cascade behavior (changing model clears runtime, etc.)
4. Test framework overhead measurement (direct vs wrapper comparison)
5. Update documentation (index.md, architecture.md)

CDN sources for models¶

Thesis benchmark combinations¶

With all 3 runtimes × 2 modes (direct + framework wrapper), the prototype produces these benchmark rows:

For each combination, the user selects: model (MobileNet v2, SqueezeNet, or EfficientNet-Lite0) and backend (WASM, WebGPU, etc.), then runs inference on an input image.