emscripten-core / emscripten-core/emscripten

Multithreading worker issue

Open
#21,959 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
27.6k
Forks
3.6k
Avg merge
1d 1h
Merged PRs (30d)
105

Description

V.3.1.39

I am compiling and running a program which use 3rd party C++ library.

the 3rd party library uses multithreading.
theoretically uses the number of processors as maximum number of threads, (that number is an input I give using `std::thread::hardware_concurrency();`)

but when this program is compiled with Emscripten and run in the browser, seems that it creates/uses a different web worker for each iteration, instead of "reusing" already created threads/workers, as it should.

this is where multithreading starts` init_future_ = std::async(std::launch::async, [this]() { process_concurrently(); });`:

```
bool initialize(){
...
if (num_threads_ != 1) {
collect();

init_future_ = std::async(std::launch::async, [this]() { process_concurrently(); });

// wait for the first element, because after init(), get() can be called.
// so the element conversion must succeed
initialization_outcome_ = wait_for_element();
} else {
initialization_outcome_ = create();
}
...
}
```

this is processConcurrently function

```
void process_concurrently() {
size_t conc_threads = num_threads_;
if (conc_threads > tasks_.size()) {
conc_threads = tasks_.size();
}

kernel_pool.reserve(conc_threads);
for (unsigned i = 0; i < conc_threads; ++i) {
kernel_pool.push_back(new MAKE_TYPE_NAME(Kernel)(kernel));
}

std::vector> threadpool;

for (auto& rep : tasks_) {
MAKE_TYPE_NAME(Kernel)* K = nullptr;
if (threadpool.size() < kernel_pool.size()) {
K = kernel_pool[threadpool.size()];
}

while (threadpool.size() == conc_threads) {
for (int i = 0; i < (int)threadpool.size(); i++) {
auto& fu = threadpool[i];
std::future_status status;
status = fu.wait_for(std::chrono::seconds(0));
if (status == std::future_status::ready) {
process_finished_rep(fu.get());

std::swap(threadpool[i], threadpool.back());
threadpool.pop_back();
std::swap(kernel_pool[i], kernel_pool.back());
K = kernel_pool.back();
break;
} // if
} // for
} // while

std::future fu = std::async(
std::launch::async, [this](
IfcGeom::MAKE_TYPE_NAME(Kernel)* kernel,
const IfcGeom::IteratorSettings& settings,
geometry_conversion_task* rep) {
this->create_element_(kernel, settings, rep);
return rep;
},
K,
std::ref(settings),
&rep);

if (terminating_) {
break;
}

threadpool.emplace_back(std::move(fu));
}

for (auto& fu : threadpool) {
process_finished_rep(fu.get());
}

finished_ = true;

if (!terminating_) {
Logger::Status("\rDone creating geometry (" + boost::lexical_cast(all_processed_elements_.size()) +
" objects) ");
}
}
```

I think this could be somehow related to [https://github.com/emscripten-core/emscripten/issues/8201](this)

ps. I tried setting THREAD_POOL_SIZE to 490 (just as a test) and the iteration goes for more or less 490 objects.

this is an example on how I use the iterator

```
if (geom_iterator_serializer.initialize()) {
do {

// Do my stuff
//Here seems that every iteration uses a new thread/worker
//leading to errors
} while (geom_iterator_serializer.next());
}
```

errors:

```
Tried to spawn a new thread, but the thread pool is exhausted.
This might result in a deadlock unless some threads eventually exit or the code explicitly breaks out to the event loop.
If you want to increase the pool size, use setting `-sPTHREAD_POOL_SIZE=...`.

worker.js onmessage() captured an uncaught exception: std::__2::system_error: thread constructor failed: Resource temporarily unavailable

Uncaught exception from main loop: ffbd51c6-1c0a-44cc-b833-e08ba3ca66c0:9:99729

unwind

Halting program

Pthread 0x4c147998 sent an error! blob:http://localhost:58010/ffbd51c6-1c0a-44cc-b833-e08ba3ca66c0:9: Error: thread constructor failed: Resource temporarily unavailable
```

any tip will be much appreciated!!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.