commaai / commaai/flash

Speed up Flash

Open
#101 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
20
Forks
73
PR merge metrics
No merged PRs in 30d

Description

- [ ] The download of images from Azure is slow - takes about 2.5 min for 900MB system image (today 4.5m...)
- [x] download/unpack in parallel (if both download and unpack take 4m, we have saved 4m)
- [x] replace `js-sha` with [`hash-wasm`](https://www.npmjs.com/package/hash-wasm) (unpack 10m -> 4m)
- updating the sha256 hash for each chunk during unpack is slow - seems to take 6ms for every chunk
- [ ] `xz-decompress` is slow
- [ ] There may be overhead because for the XZ decompression and SHA calculation we transfer bytes from JS to WASM (and back for decompression)... Not sure yet how complex it is to write something to process everything on the WASM side.
- Moving the `QdlManager` to a web worker (instead of just `ImageWorker`) may also help speed things up as it would move the flashing (which includes computations like `Sparse.splitBlob`) off of the main/render thread
- Faster flashing of sparse images in `qdl.js` (flashing from 2:30 to 1:50)
- [x] fast erase partition/sector
- [x] skip chunks during flashing
- [ ] Stream images to flash while downloading (`qdl.js` needs to accept a `ReadableStream` for blob to flash)

You can generate perf traces in Chrome in the performance tab

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with qdl.js, QdlManager, ImageWorker, and Sparse.splitBlob, then use Chrome's Performance tab to reproduce the reported flashing and download timings. The issue lists several possible optimization paths, including streaming downloads and moving work off the render thread; done would require selecting and implementing a scoped improvement with measurable performance gains.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, javascript, wasm
Domain
performance, web-dev
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.