emscripten-core / emscripten-core/emscripten

Design issues with A new filesystem backend which support mount files and work properly in multithreaded app based on Pthreads API

Open
#20,912 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
27.6k
Forks
3.6k
Avg merge
1d 1h
Merged PRs (30d)
105

Description

Hello,

I'm working on a multithreaded Emscripten application (based on the POSIX Threads API) that requires all non-main threads to support reading file objects obtained through the browser's showOpenFilePicker API. I have thoroughly reviewed the File System API, including WasmFS, but have not found a solution that gets the job done.

To better clarify my requirements, I'll describe them again using pseudocode.

Let's assume there's a new file system called PTHREADFS, which provides read-only access to File and Blob objects without copying the entire data into memory and can potentially be used for huge files.

Using the browser's showOpenFilePicker API, we'll select a file named test.txt and obtain the corresponding files object. Then, we'll mount the selected files object in the main thread:

```
FS.mkdir('/work');
FS.mount(PTHREADFS, { files: files }, '/work');
```

After that, I expect to be able to read the file content through the file path in multiple other non-main threads, like this:

```
std::thread task1(readFileContent, "/work/test.txt");
std::thread task2(readFileContent, "/work/test.txt");

void readFileContent(const std::string& filePath) {
std::ifstream fileStream(filePath);
if (!fileStream.is_open()) {
std::cerr << "Failed to open the file " << filePath << std::endl;
return;
}

std::string line;
while (std::getline(fileStream, line)) {
std::cout << line << std::endl;
}

fileStream.close();
}
```

Since the JavaScript-based file system must proxy to the main thread where all filesystem operations are done, I'm wondering if it would be feasible for all other threads to proxy file reading to the main thread? Then, the main thread could read the file through the asynchronous API of FileReader (which wouldn't block the main thread).

In addition, would it be feasible to implement a new backend similar to opfs backend based on WasmFS, start a separate worker thread to handle file reading operations which read the file through the synchronous API of FileReaderSync, and then proxy all other threads' file reading operations to the worker thread instead of the main thread? Under these circumstances, how can the files object obtained from showOpenFilePicker in the main thread be elegantly passed to the worker thread?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.