emscripten-core / emscripten-core/emscripten

MEMFS does not support files >2gb

Open
#20,245 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
27.6k
Forks
3.6k
Avg merge
1d 1h
Merged PRs (30d)
105

Description

Currently there is no way to create a file >2gb since you cannot create an ArrayBuffer >2gb from JS
But you can create WebAssembly.Memory with arbitrary length, get it's buffer and substitute FS node contents with a UInt8Array. But that leads to a 4gb limit of UInt8Array

So i'd like to ask how to implement a fix in a better way:
1. Add a check in memfs allocate function and create WebAssembly.Memory to fit the new size if it's >2gb and then
a. create a proxy for UInt8Array for contents just like with memory access
b. fix everything to work with ArrayBuffer directly (not sure what the issue is but it just returns zeroes when mmap is used)
2. Add a function to create a file with ArrayBuffer created by user and fix issues with memfs and ArrayBuffer
3. Same as 2 but add an option to store the file in main wasm memory. This will resolve an issue with memory usage doubling for mmap but the file cannot be resized (i think it's ok)

I think 3 is most efficient and won't need a fix for chrome to lift memory limit on a tab from 16gb. I need to load 7gb file and mmap it, so avoiding memory doubling would help a lot

@sbc100 what do you think?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.