emscripten-core / emscripten-core/emscripten
[WasmFS+OPFS] Unspecified file size limit
- Dominant language
- C++
- Stars
- 27.6k
- Forks
- 3.6k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 105
Description
I have a wasm file compiled with `-lopfs.js -sWASMFS -sFORCE_FILESYSTEM` and I'm trying to perform IO operations on a large (4GB) file. The process works properly for smaller files (~150MB, probably anything lower than `2^32-1` bytes). However, I've noticed that when I try to read this large file from WASM it fails. When I try to read it using `readFile` from JS I don't see correct size on ArrayBuffer. I don't expect to see all of them (ArrayBuffer limit) but it's way less than my browser's limit (Chrome).
In essence, I'm trying to pass a `File` provided by user to `OPFS` and use it with my executable. `WORKERFS` works very nice if I create the node with `WORKERFS.createNode`. However, the problem with that is reading back the data if it's larger than `2^32-1` bytes. I'm considering overloading `WORKERFS.stream_ops.write`/`FS.write` to write to `OPFS` and create `File` object using that.
> I've verified that the file that has been written to OPFS was correct by downloading it again.
A wild guess but maybe **seeking more than `2^32-1` bytes** at once is creating this problem?
EDIT: I've confirmed this is the problem. I had a MP4 file intentionally made larger than it's supposed to be. This is done by marking first sample at an offset of ~4GB. Then I've tried a normal 4GB downloaded from YouTube and it worked properly. So it's not size issue but rather seeking issue.
What's your opinion on this process? Is there really a file size limit with `OPFS` or `WasmFS`? What about monkey-patching `WORKERFS`?
Contributor guide
Assessment
This issue has not been assessed yet.