emscripten-core / emscripten-core/emscripten
MEMFS does not support files >2gb
- Dominant language
- C++
- Stars
- 27.6k
- Forks
- 3.6k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 105
Description
Currently there is no way to create a file >2gb since you cannot create an ArrayBuffer >2gb from JS
But you can create WebAssembly.Memory with arbitrary length, get it's buffer and substitute FS node contents with a UInt8Array. But that leads to a 4gb limit of UInt8Array
So i'd like to ask how to implement a fix in a better way:
1. Add a check in memfs allocate function and create WebAssembly.Memory to fit the new size if it's >2gb and then
a. create a proxy for UInt8Array for contents just like with memory access
b. fix everything to work with ArrayBuffer directly (not sure what the issue is but it just returns zeroes when mmap is used)
2. Add a function to create a file with ArrayBuffer created by user and fix issues with memfs and ArrayBuffer
3. Same as 2 but add an option to store the file in main wasm memory. This will resolve an issue with memory usage doubling for mmap but the file cannot be resized (i think it's ok)
I think 3 is most efficient and won't need a fix for chrome to lift memory limit on a tab from 16gb. I need to load 7gb file and mmap it, so avoiding memory doubling would help a lot
@sbc100 what do you think?
Contributor guide
Assessment
This issue has not been assessed yet.