emscripten-core / emscripten-core/emscripten

WasmFS: global init + fetch backend problem

Open
#18,475 8 comments 0 reactions 0 assignees View on GitHub
wasmfs
Dominant language
C++
Stars
27.6k
Forks
3.6k
Avg merge
1d 1h
Merged PRs (30d)
105

Description

What is a correct way to initialize the FS backend so that files from it can be read during the global initialization? Global initialization happens in the main browser thread. wasmfs_before_preload is called during global init, so we can’t create fetch backend in it. Even if compiled with -sPROXY_TO_PTHREAD=1 global init happens in the main browser thread while main() runs in a different thread.
Creating backend inside the main function with sPROXY_TO_PTHREAD is too late.

**Version of emscripten/emsdk:**

```
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.30-git (0d7014e1b20ead9739f1fce54b316152413601c3)
clang version 16.0.0 (https://github.com/llvm/llvm-project 87f57f459e7acbb00a6ca4ee6dec6014c5a97e07)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: C:\Users\kdots\emsdk\3.1.27\upstream\bin
```

**Failing command line in full:**

```
#include
#include
#include
#include

const char *filePath = "/test.txt";
const char *url = "http://localhost:8000/test.txt";

struct S
{
S()
{
char buffer[100] = {'\0'};
auto* fp = fopen(filePath, "rb");
assert(fp);
assert(fread(buffer, 1, 13, fp) > 0);
printf("%s", buffer);
fclose(fp);
}
};

S s;

void init_backend()
{
backend_t backend = wasmfs_create_fetch_backend(url);
int fd = wasmfs_create_file(filePath, S_IRWXUGO, backend);
assert(fd > 0);
close(fd);
}

void wasmfs_before_preload()
{
init_backend(); // [1]
}

int main()
{
init_backend(); // [2]
return 0;
}
```

**Full link command and output with `-v` appended:**



```
C:\emscripten>em++ -std=c++17 C:/build/test.cpp -o C:/build/test.html -s FORCE_FILESYSTEM=1 -s EXIT_RUNTIME=1 -s WASMFS=1 -s USE_PTHREADS=1 -sPTHREAD_POOL_SIZE=4 -sPROXY_TO_PTHREAD=1 -v
"C:/Users/kdots/emsdk/3.1.27/upstream/bin\clang.exe" --version
"C:/Users/kdots/emsdk/3.1.27/upstream/bin\clang++.exe" -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_SHARED_MEMORY__=1 -DEMSCRIPTEN --sysroot=C:\emscripten\cache\sysroot -Xclang -iwithsysroot/include\fakesdl -Xclang -iwithsysroot/include\compat -std=c++17 -v -pthread C:/build/test.cpp -c -o C:\Users\kdots\AppData\Local\Temp\emscripten_temp_pan3ej_z\test_0.o
clang version 16.0.0 (https://github.com/llvm/llvm-project 87f57f459e7acbb00a6ca4ee6dec6014c5a97e07)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: C:\Users\kdots\emsdk\3.1.27\upstream\bin
(in-process)
"C:\\Users\\kdots\\emsdk\\3.1.27\\upstream\\bin\\clang++.exe" -cc1 -triple wasm32-unknown-emscripten -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -main-file-name test.cpp -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-feature +atomics -target-feature +bulk-memory -target-feature +mutable-globals -target-feature +sign-ext -target-cpu generic -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v "-fcoverage-compilation-dir=C:\\emscripten" -resource-dir "C:\\Users\\kdots\\emsdk\\3.1.27\\upstream\\lib\\clang\\16" -D __EMSCRIPTEN_SHARED_MEMORY__=1 -D EMSCRIPTEN -isysroot "C:\\emscripten\\cache\\sysroot" -internal-isystem "C:\\emscripten\\cache\\sysroot/include/wasm32-emscripten/c++/v1" -internal-isystem "C:\\emscripten\\cache\\sysroot/include/c++/v1" -internal-isystem "C:\\Users\\kdots\\emsdk\\3.1.27\\upstream\\lib\\clang\\16\\include" -internal-isystem "C:\\emscripten\\cache\\sysroot/include/wasm32-emscripten" -internal-isystem "C:\\emscripten\\cache\\sysroot/include" -std=c++17 -fdeprecated-macro "-fdebug-compilation-dir=C:\\emscripten" -ferror-limit 19 -fmessage-length=192 -fvisibility=default -pthread -fgnuc-version=4.2.1 -fcxx-exceptions -fignore-exceptions -fexceptions -fcolor-diagnostics "-iwithsysroot/include\\fakesdl" "-iwithsysroot/include\\compat" -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o "C:\\Users\\kdots\\AppData\\Local\\Temp\\emscripten_temp_pan3ej_z\\test_0.o" -x c++ C:/build/test.cpp
clang -cc1 version 16.0.0 based upon LLVM 16.0.0git default target x86_64-pc-windows-msvc
ignoring nonexistent directory "C:\emscripten\cache\sysroot/include/wasm32-emscripten/c++/v1"
ignoring nonexistent directory "C:\emscripten\cache\sysroot/include/wasm32-emscripten"
#include "..." search starts here:
#include <...> search starts here:
C:\emscripten\cache\sysroot/include\fakesdl
C:\emscripten\cache\sysroot/include\compat
C:\emscripten\cache\sysroot/include/c++/v1
C:\Users\kdots\emsdk\3.1.27\upstream\lib\clang\16\include
C:\emscripten\cache\sysroot/include
End of search list.
"C:/Users/kdots/emsdk/3.1.27/upstream/bin\wasm-ld.exe" -o C:/build/test.wasm C:\Users\kdots\AppData\Local\Temp\emscripten_temp_pan3ej_z\test_0.o -LC:\emscripten\cache\sysroot\lib\wasm32-emscripten C:\emscripten\cache\sysroot\lib\wasm32-emscripten\crtbegin.o C:\emscripten\cache\sysroot\lib\wasm32-emscripten\crt1_proxy_main.o --whole-archive -lwasmfs-mt-debug --no-whole-archive -lGL-mt -lal -lhtml5 -lstubs-debug -lc-mt-debug -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-debug-mt-noexcept -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --allow-undefined-file=C:\Users\kdots\AppData\Local\Temp\tmpnm8ka7dp.undefined --import-memory --shared-memory --strip-debug --export-if-defined=main --export-if-defined=_emscripten_thread_init --export-if-defined=_emscripten_thread_exit --export-if-defined=_emscripten_thread_crashed --export-if-defined=_emscripten_tls_init --export-if-defined=pthread_self --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=__main_argc_argv --export-if-defined=fflush --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_get_current --export=emscripten_stack_init --export=_wasmfs_read_file --export=_wasmfs_write_file --export=_wasmfs_mkdir --export=_wasmfs_unlink --export=_wasmfs_chdir --export=_wasmfs_symlink --export=_wasmfs_chmod --export=_wasmfs_identify --export=_wasmfs_readdir_start --export=_wasmfs_readdir_get --export=_wasmfs_readdir_finish --export=stackSave --export=stackRestore --export=stackAlloc --export=__wasm_call_ctors --export=__errno_location --export=emscripten_dispatch_to_thread_ --export=_emscripten_thread_free_data --export=emscripten_main_browser_thread_id --export=emscripten_main_thread_process_queued_calls --export=emscripten_run_in_main_runtime_thread_js --export=emscripten_stack_set_limits --export=emscripten_proxy_finish --export=emscripten_proxy_execute_queue --export=__get_temp_ret --export=__set_temp_ret --export=__funcs_on_exit --export=malloc --export=free --export-table -z stack-size=65536 --initial-memory=16777216 --entry=_emscripten_proxy_main --max-memory=16777216 --stack-first
"C:/Users/kdots/emsdk/3.1.27/upstream\bin\wasm-emscripten-finalize" --dyncalls-i64 --pass-arg=legalize-js-interface-exported-helpers C:/build/test.wasm -o C:/build/test.wasm --detect-features
"C:\Users\kdots\emsdk\3.1.27\node\14.18.2_64bit\bin\node.exe" C:\emscripten\src\compiler.js C:\Users\kdots\AppData\Local\Temp\tmpk15ipler.json
"C:/Users/kdots/emsdk/3.1.27/upstream/bin\llvm-objcopy.exe" C:/build/test.wasm C:/build/test.wasm --remove-section=.debug* --remove-section=producers
"C:\Users\kdots\emsdk\3.1.27\node\14.18.2_64bit\bin\node.exe" C:\emscripten\tools\preprocessor.js C:\Users\kdots\AppData\Local\Temp\emscripten_temp_pan3ej_z\settings.js worker.js --expandMacros
"C:\Users\kdots\emsdk\3.1.27\node\14.18.2_64bit\bin\node.exe" C:\emscripten\tools\preprocessor.js C:\Users\kdots\AppData\Local\Temp\emscripten_temp_pan3ej_z\settings.js shell.html
```

Output in the first case when only wasmfs_preload is called:
```
test.js:1088 Aborted(Assertion failed: !emscripten_is_main_browser_thread() && "Cannot safely create fetch backend on main browser thread", at: C:\emscripten\system\lib\wasmfs\backends\fetch_backend.cpp,90,wasmfs_create_fetch_backend)
```
Output in the second case when only main is called:
```
Uncaught RuntimeError: Aborted(Assertion failed: fp, at: C:/build/test.cpp,15,S)
at abort (test.js:1107:11)
at ___assert_fail (test.js:1849:7)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.