Benchmarking Plan¶

Three-dimensional benchmarking methodology Status note: the three-dimension framing below is superseded by the implemented three-axis methodology in research/plans/impl-plan-benchy.md (§ Methodology alignment): Axis 1 = strict CPU/wasm baseline, Axis 2 = measured GPU acceleration benefit, Axis 3 = runtime efficiency vs. theoretical-max coverage. The frozen suite (30 runs / 5 warmup, wasm baseline per runtime) lives in benchy/src/core/suiteConfig.ts. Statistical analysis (e.g., Mann-Whitney U) is computed offline from the raw-sample exports — benchy records and exports per-iteration samples but does not run significance tests in-app. The dimension tables below remain useful as the conceptual origin of the axis design.

Overview¶

The benchmarking study answers one core question: Is it worth using high-level ML frameworks, or should developers go straight to the runtime?

Three dimensions, each isolating a different aspect of web ML performance:

Dimension	Question	What it proves
1 — Low-level engines	Which runtime is fastest?	Runtime quality comparison
2 — High-level APIs	Which framework is fastest?	Framework quality comparison
3 — Abstraction overhead	What does the framework add?	Whether abstraction is worth its cost

Dimension 1 — Low-level engines (head-to-head)¶

Runtimes compared: TensorFlow.js vs ONNX Runtime Web vs LiteRT.js

Method: Each runtime runs the same model (MobileNet v2) with identical input, measuring:

Metric	How measured
Cold start time	Model download + compile + first inference
Warm inference latency	Single inference after warmup (p50, p95, p99)
Throughput	Inferences per second (batch size 1)
Memory usage	Peak JS heap + WASM memory
Bundle size	Runtime WASM + JS payload

Backends tested per runtime:

Runtime	WASM	WebGL	WebGPU
TF.js	✅	✅	✅
ORT Web	✅	⚠️ deprecated	✅
LiteRT.js	✅	❌	✅

Model: MobileNet v2 (same architecture, different formats):

Runtime	Model format	Source
TF.js	TF.js graph	TF Hub (jsDelivr)
ORT Web	ONNX	HuggingFace Hub
LiteRT.js	.tflite	HuggingFace / TF Hub

Expected outcome: LiteRT.js fastest (native C++), ORT Web close, TF.js slower (JS kernels).

Dimension 2 — High-level APIs (head-to-head)¶

Frameworks compared: Transformers.js vs MediaPipe Tasks vs ML5.js

Method: Each framework runs its equivalent task with identical input:

Framework	Task	Model	Metric
Transformers.js	Image classification	MobileNet v2 (ONNX)	Pipeline latency
MediaPipe Tasks	Image classification	EfficientNet-Lite0	Task runner latency
ML5.js	Image classification	MobileNet v2 (TF.js)	Classification latency

Metrics: Same as Dimension 1 (cold start, warm latency, throughput, memory).

Key difference: Each framework manages its own model loading, preprocessing, and postprocessing. The measurement captures the full end-to-end pipeline, not just inference.

Expected outcome: MediaPipe fastest (task-optimized, minimal abstraction), Transformers.js middle, ML5.js slowest (thinnest wrapper but still adds overhead).

Dimension 3 — Abstraction overhead¶

Question: For each framework→runtime pair, how much slower is the framework compared to using the runtime directly?

Pairings:

High-level	Wraps	Comparison
ML5.js	TensorFlow.js	ML5 classify() vs TF.js model.predict()
Transformers.js	ORT Web	Transformers pipeline('image-classification') vs ORT Web InferenceSession.run()
MediaPipe Tasks	LiteRT.js	MediaPipe ImageClassifier vs LiteRT.js model.run()

Method: For each pair:

Run the same model via the framework (high-level API)
Run the same model via the underlying runtime (low-level API)
Subtract: overhead = framework_latency − runtime_latency
Express as percentage: overhead_pct = (overhead / runtime_latency) × 100

What each pairing reveals:

Pairing	What overhead includes
ML5.js → TF.js	ML5 API abstraction + model loading wrapper
Transformers.js → ORT Web	Tokenization + preprocessing + pipeline orchestration + ONNX session management
MediaPipe Tasks → LiteRT.js	Task-specific preprocessing + postprocessing + model lifecycle management

Expected outcome: Frameworks add 5–30% overhead depending on abstraction depth. The thesis evaluates whether this overhead is justified by developer experience gains.

Test environment¶

Parameter	Value
Browser	Chrome (latest stable)
Hardware	Mid-range laptop (integrated GPU) + desktop (discrete GPU)
Network	Throttled to 3G for cold start tests; uncached for warm tests
Warmup	5 iterations before measurement (updated, was 10 — frozen protocol A8/A19)
Sample size	30 measured runs per configuration (updated, was 100 — A8 minimum; ≥30 per Georges et al., OOPSLA 2007)
Statistics	Median (p50) + p95 over measured runs only, warmups excluded (updated; Mann-Whitney U dropped with the frozen suite)

Model distribution (final)¶

Runtime/Framework	Model	Format	Source
TF.js	MobileNet v2	TF.js graph	TF Hub
ORT Web	MobileNet v2	ONNX	HuggingFace Hub
LiteRT.js	MobileNet v2	.tflite	HuggingFace / TF Hub
Transformers.js	MobileNet v2	ONNX	HuggingFace Hub
MediaPipe Tasks	EfficientNet-Lite0	.tflite	Google Storage
ML5.js	MobileNet v2	TF.js graph	TF Hub

Model parity

All three runtimes benchmark MobileNet v2 for direct comparison. MediaPipe uses EfficientNet-Lite0 because that's its pre-optimized model — comparing it separately shows task-specific optimization vs general-purpose inference.

Deliverables per dimension¶

Dimension	Deliverable	Format
1	Runtime comparison table	Latency/throughput/memory per runtime×backend
2	Framework comparison table	End-to-end pipeline performance per framework
3	Overhead analysis	Absolute overhead (ms) + relative overhead (%) per pair
All	Statistical significance	Mann-Whitney U p-values for all pairwise comparisons
All	Recommendations	"Use X when Y" decision matrix for developers