openzim / openzim/javascript-libzim

Extremely slow file handling with archives in Android

Open
#42 8 comments 0 reactions 1 assignee View on GitHub

@Jaifroid is already working on this.

Since Jan 8, 2023.

bug help wanted
Dominant language
JavaScript
Stars
5
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Issue was initially discussed and diagnosed over at https://github.com/kiwix/kiwix-js-windows/issues/343. There, I thought the issue was with Emscripten's WORKERFS. However, using the test case in this Repo for large file access, and debugging on a Chromium instance on a midrange Samsung Android, the WORKERFS has no problem loading and reading (instantly) a 92GB Wikipedia ZIM from a microSD card. See screenshot below. If we can read bytes from the end of the file nearly instantly, why is javascript-libzim getting such awful performance? Loading Ray Charles into the WASM on Android takes 30 seconds in Samsung Internet (a Chromium browser), and nearly 90 seconds in Chrome. I was unable to load anything larger than 500MB into either instance.

The slowdown is mostly in relation to instantiating the archive. Once the archive has registered, full-text searching is reasonable. This is why I initially thought the issue was to do with passing the ArrayBuffer to the Web Worker, but big file test does this instantly. @mgautierfr would you have any thoughts on what could be going on here?

image

NB we can't test on Firefox because Firefox on Android attempts to copy picked archives into memory (or possibly an internal file system), and crashes on anything larger than about 2GB.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.