GoogleChrome / GoogleChrome/webai-compute-benchmark

Huge total network transfers (due to model weights mostly, but also some caching issue)

Open
#33 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
13
Forks
5
PR merge metrics
No merged PRs in 30d

Description

I just ran the current main branch locally in Chrome and looked at the DevTools -> Network tab, since the first iteration took quite a while to run (>10 minutes).

In total, we are transferring >2GB of data. See screenshot from the status bar of the network tab:
Image

Clearly, the biggest reason are large ONNX runtime model weights, see this screenshot of the requests by size. However, some Wasm model (I think the transformers-js one) is also repeatedly downloaded:
Image

This is going to be an issue for a couple of reasons:
- first iteration is super slow and shows no indication of progress or what it's doing
- users might get surprised/angry of data usage over metered connections
- users might run into OOMs or full disk space
- possibly hosting costs for us

In terms of solutions, I think we should (roughly in order of urgency / ease of implementation)
- [ ] Before starting the benchmark / on the status page, show a warning sentence, something like "Warning: Running this benchmark downloads large models and incurs network transfers in the order of 2GB of data. Only run with a fast network and not over a metered connection."
- [ ] Fix the duplicate download of Wasm model files (possibly related: #10)
- [ ] Select smaller models, in particular for Transformers.js
- [ ] Can we download the models on the client ahead-of-time, before any iteration?

Contributor guide

Open the contributing guide

Research direction

Reproduce the report by running the current main branch in Chrome and inspecting the DevTools Network tab during the first iteration; the issue points to the benchmark and status page but names no files or tests. First separate repeated Wasm downloads from model-weight transfers, then define completion as an implemented warning or a verified fix for one selected cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, wasm
Domain
performance, web-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.