Model & Task Distribution¶

Full task catalog, model sourcing, and dataset planning Revised after verification (Context7 + Firecrawl + benchy validation gate): runtime-support claims corrected, core benchmark restructured, settings aligned with the frozen benchy suite

Overview¶

The platform provides a comprehensive ML test suite spanning computer vision, audio, and NLP. Each task includes standard benchmark datasets, multiple model options in web-friendly formats, and a runtime/framework compatibility matrix.

The core benchmark automates fair comparisons across runtimes using matched models. Users can also run any task manually with custom configurations and run counts (1, 10, 25, 50, or more; below 30 measured runs benchy draws a statistical warning — automated sweeps clamp to the 30-run minimum, A8).

Task Categories¶

Core Benchmark¶

The core benchmark is an automated "autoplay" comparison that runs matched models across all compatible runtimes with standardized settings.

Inclusion criteria:

1. Same model architecture available in ≥3 formats (ONNX, TFLite, TF.js) — only image classification meets this today
2. Standard benchmark dataset with published metrics
3. Automatable evaluation (no subjective judgment)
4. Supported by ≥3 runtimes/frameworks

Core benchmark tasks:

Qualifying candidates for future auto-bench tasks (omni-task roadmap):

Key verified facts:

• No identical detection model exists across runtimes. ONNX zoo ships SSD-MobileNetV1 (not V2); TF.js coco-ssd bases are lite_mobilenet_v2/v1/v2; MediaPipe uses EfficientDet-Lite0; Transformers.js only supports RT-DETR/YOLOS/DETR for detection.
• Input resolution is a model property, not a runtime property (coco-ssd ~300², EfficientDet-Lite0 320², MoveNet 192²/256², selfie-multiclass 256²). It must be recorded per run and treated as a documented model attribute.
• LiteRT.js I/O constraint: int32/float32 tensors only — this rules out uint8-quantized pose models and int8 detection models for LiteRT.
• Post-processing location differs per engine: TF.js coco-ssd and MediaPipe ship post-processed outputs; ONNX zoo SSD ships raw boxes/scores (JS decode + NMS needed). This is a documented validity factor.
• Future extension — converted same-model pairs: converting one model (e.g., RTMDet-n, RTMPose-t) to both ONNX and TFLite via litert-torch/onnx2tf enables a strict ORT↔LiteRT comparison for detection/pose. Requires one-time conversion + vendoring effort. See Model Sources & Conversion.

Core benchmark settings (aligned with the frozen benchy suite):

• Warmup: 5 iterations
• Measurement runs: 30 (frozen in suiteConfig.ts; A8 minimum clamp applies)
• Statistical analysis (e.g., Mann-Whitney U): computed offline from the raw-sample exports, not in-app
• Network: uncached (models vendored same-origin; cold-start = first load, reported separately via firstInferenceMs)

Computer Vision Tasks¶

1. Image Classification ★ CORE BENCHMARK¶

Classify the major object in an image into a set of pre-defined classes.

Dataset: ImageNet-1K (ILSVRC 2012) — 50K validation images, 1000 classes

Metric: Top-1 Accuracy, Top-5 Accuracy, Inference Latency (ms)

Models:

Runtime Support:

2. Object Detection ★ CORE BENCHMARK¶

Detecting instances of semantic objects of a certain class in digital images and videos, outputting bounding boxes and labels.

Dataset: COCO val2017 — 5K images, 80 classes

Metric: [email protected], [email protected]:0.95, Inference Latency (ms)

Models:

Runtime Support:

3. Semantic Segmentation ★ CORE BENCHMARK¶

Partitioning an image into semantically meaningful parts, classifying each pixel into a pre-determined class.

Dataset: PASCAL VOC 2012 — 1.4K validation images, 20 classes + background

Metric: mIoU (mean Intersection over Union), Inference Latency (ms)

Models:

Runtime Support:

4. Pose Estimation ★ CORE BENCHMARK¶

Detect human body keypoints (skeleton) — joints like shoulders, elbows, knees, and ankles.

Dataset: COCO Keypoints val2017 — 5K images, 17 keypoints

Metric: AP (Average Precision), Inference Latency (ms)

Models:

Runtime Support:

5. Facial Landmark Detection ★ CORE BENCHMARK¶

Detecting facial landmarks like eyes, nose, mouth, and jawline — used for face alignment, tracking, and try-on applications.

Dataset: 300W — 68 landmarks; WFLW — 98 landmarks

Metric: NME (Normalized Mean Error), Inference Latency (ms)

Models:

Runtime Support:

6. Face Recognition¶

Detecting faces and verifying whether a face matches a known identity.

Dataset: LFW (Labeled Faces in the Wild) — 13K images, 5.7K identities

Metric: Verification Accuracy, TAR @ FAR=0.001, Inference Latency (ms)

Models:

7. Body Segmentation¶

Segmenting human body regions from the background — used for background blur, virtual backgrounds, and AR effects.

Dataset: COCO-Stuff / ADE20K

Metric: mIoU, Inference Latency (ms)

Models:

8. Depth Estimation¶

Predicting depth from a single image — used for 3D reconstruction, AR placement, and scene understanding.

Dataset: DIODE / NYU Depth V2

Metric: Abs Rel, RMSE, δ < 1.25, Inference Latency (ms)

Models:

9. Image Style Transfer¶

Applying artistic styles (e.g., Van Gogh, Monet) to photographs in real time.

Dataset: WikiArt (perceptual quality — no standard accuracy metric)

Metric: Inference Latency (ms), FID (optional)

Models:

Audio Tasks¶

10. Noise Suppression¶

Removing background noise from audio in real time — used for video calls, podcasts, and voice assistants.

Dataset: DNS Challenge (Microsoft Deep Noise Suppression Challenge)

Metric: PESQ (Perceptual Evaluation of Speech Quality), STOI, Inference Latency (ms)

Models:

11. Speech Recognition (ASR)¶

Converting spoken language into text — used for transcription, voice commands, and accessibility.

Dataset: LibriSpeech / CommonVoice

Metric: WER (Word Error Rate), Inference Latency (ms)

Models:

12. Audio Classification¶

Classifying audio clips into categories — used for sound event detection, environmental monitoring, and music genre classification.

Dataset: ESC-50 / AudioSet

Metric: Accuracy, Inference Latency (ms)

Models:

NLP Tasks¶

13. Text Classification¶

Assigning a label or class to text — used for sentiment analysis, spam detection, topic categorization, and toxicity detection.

Dataset: GLUE Benchmark / SST-2 (Stanford Sentiment Treebank)

Metric: Accuracy, GLUE Score, Inference Latency (ms)

Models:

14. Text Embeddings¶

Generating dense vector representations of text — used for semantic search, clustering, and retrieval-augmented generation (RAG).

Dataset: MTEB (Massive Text Embedding Benchmark)

Metric: MTEB Average Score, Inference Latency (ms)

Models:

15. Question Answering¶

Extracting answers from a context paragraph — used for document search, chatbots, and knowledge bases.

Dataset: SQuAD v2.0 (Stanford Question Answering Dataset)

Metric: F1 Score, Exact Match, Inference Latency (ms)

Models:

16. Text Generation¶

Generating text autoregressively given a prompt — used for content creation, code generation, and conversational AI.

Dataset: WikiText-2 (perplexity)

Metric: Perplexity, Inference Latency (tokens/sec)

Models:

Cross-Reference: Runtime × Task Availability¶

Verified rows — checked against official docs, model hubs, and the benchy validation gate:

Remaining catalog rows (unverified estimates) — the audio and NLP rows below were drafted optimistically and have not been verified the way the rows above were. Treat them as leads, not facts:

Model Sources¶

Model Sources & Conversion¶

Model availability in a browser-executable format is itself a compatibility property. The conversion toolchain from creation frameworks to browser formats:

Taxonomy (mirrors thesis Background §Conversion Paths):

• Creation/training frameworks: PyTorch, TensorFlow/Keras, JAX
• Interchange format: ONNX
• Edge optimization/deployment runtimes (not creation frameworks): OpenVINO, TensorRT, LiteRT
• Legacy: Caffe (historical training framework)

Implications for the benchmark:

1. Format choice gates runtime choice — a model only exists for the runtimes whose format it was converted to
2. Conversion settings (opset, quantization, NCHW/NHWC) are part of the experimental setup and must be documented per artifact
3. Conversion is the precondition for strict same-model cross-runtime comparison — the MobileNetV2 artifacts in public/models/ exist in three formats precisely because of this toolchain
4. Future extension: converting one detection/pose model (e.g., RTMDet-n, RTMPose-t) to both ONNX and TFLite would enable a strict ORT↔LiteRT pair beyond classification (one-time conversion + vendoring effort; fp16 TFLite keeps float32 I/O and satisfies the LiteRT.js constraint)

User Workflows¶

The platform supports two usage modes:

Automated Core Benchmark¶

Click "Start Benchmark" on the /bench page to automatically (frozen protocol, A8/A19):

1. Prepare matched models across all compatible runtimes (artifacts warmed into the Cache API so load times are cache-warm)
2. Per combo: one model load, then 5 warmup inferences pinned to the first image slot
3. Execute 30 measured inferences rotating over the 3 bench images (10 per slot; warmups never advance the rotation)
4. Per-slot majority top-1 against registered ground truths → accuracyAt1 (fraction correct)
5. Median (p50) and p95 latency stats; composite score = latency score × accuracyAt1
6. Display per-run results with verdict badges and per-run detail (ⓘ)
7. Export CSV/JSON download for plotting

Manual Task Exploration¶

Users can also:

1. Browse the full task library (16 tasks, 50+ models)
2. Select a specific task and model
3. Choose which runtime(s) to benchmark against
4. Configure run count (1, 10, 25, 50, or custom — below 30 measured runs a statistical warning is shown)
5. Run benchmarks and view detailed results
6. Upload custom models for comparison