emscripten-core / emscripten-core/emscripten

[WasmFS+OPFS] Unspecified file size limit

Open
#21,335 20 comments 0 reactions 1 assignee Claimed by @tlively View on GitHub
Dominant language
C++
Stars
27.6k
Forks
3.6k
Avg merge
1d 1h
Merged PRs (30d)
105

Description

I have a wasm file compiled with `-lopfs.js -sWASMFS -sFORCE_FILESYSTEM` and I'm trying to perform IO operations on a large (4GB) file. The process works properly for smaller files (~150MB, probably anything lower than `2^32-1` bytes). However, I've noticed that when I try to read this large file from WASM it fails. When I try to read it using `readFile` from JS I don't see correct size on ArrayBuffer. I don't expect to see all of them (ArrayBuffer limit) but it's way less than my browser's limit (Chrome).

In essence, I'm trying to pass a `File` provided by user to `OPFS` and use it with my executable. `WORKERFS` works very nice if I create the node with `WORKERFS.createNode`. However, the problem with that is reading back the data if it's larger than `2^32-1` bytes. I'm considering overloading `WORKERFS.stream_ops.write`/`FS.write` to write to `OPFS` and create `File` object using that.

> I've verified that the file that has been written to OPFS was correct by downloading it again.

A wild guess but maybe **seeking more than `2^32-1` bytes** at once is creating this problem?

EDIT: I've confirmed this is the problem. I had a MP4 file intentionally made larger than it's supposed to be. This is done by marking first sample at an offset of ~4GB. Then I've tried a normal 4GB downloaded from YouTube and it worked properly. So it's not size issue but rather seeking issue.

What's your opinion on this process? Is there really a file size limit with `OPFS` or `WasmFS`? What about monkey-patching `WORKERFS`?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.