emscripten-core / emscripten-core/emscripten
Multithreading worker issue
- Dominant language
- C++
- Stars
- 27.6k
- Forks
- 3.6k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 105
Description
V.3.1.39
I am compiling and running a program which use 3rd party C++ library.
the 3rd party library uses multithreading.
theoretically uses the number of processors as maximum number of threads, (that number is an input I give using `std::thread::hardware_concurrency();`)
but when this program is compiled with Emscripten and run in the browser, seems that it creates/uses a different web worker for each iteration, instead of "reusing" already created threads/workers, as it should.
this is where multithreading starts` init_future_ = std::async(std::launch::async, [this]() { process_concurrently(); });`:
```
bool initialize(){
...
if (num_threads_ != 1) {
collect();
init_future_ = std::async(std::launch::async, [this]() { process_concurrently(); });
// wait for the first element, because after init(), get() can be called.
// so the element conversion must succeed
initialization_outcome_ = wait_for_element();
} else {
initialization_outcome_ = create();
}
...
}
```
this is processConcurrently function
```
void process_concurrently() {
size_t conc_threads = num_threads_;
if (conc_threads > tasks_.size()) {
conc_threads = tasks_.size();
}
kernel_pool.reserve(conc_threads);
for (unsigned i = 0; i < conc_threads; ++i) {
kernel_pool.push_back(new MAKE_TYPE_NAME(Kernel)(kernel));
}
std::vector> threadpool;
for (auto& rep : tasks_) {
MAKE_TYPE_NAME(Kernel)* K = nullptr;
if (threadpool.size() < kernel_pool.size()) {
K = kernel_pool[threadpool.size()];
}
while (threadpool.size() == conc_threads) {
for (int i = 0; i < (int)threadpool.size(); i++) {
auto& fu = threadpool[i];
std::future_status status;
status = fu.wait_for(std::chrono::seconds(0));
if (status == std::future_status::ready) {
process_finished_rep(fu.get());
std::swap(threadpool[i], threadpool.back());
threadpool.pop_back();
std::swap(kernel_pool[i], kernel_pool.back());
K = kernel_pool.back();
break;
} // if
} // for
} // while
std::future fu = std::async(
std::launch::async, [this](
IfcGeom::MAKE_TYPE_NAME(Kernel)* kernel,
const IfcGeom::IteratorSettings& settings,
geometry_conversion_task* rep) {
this->create_element_(kernel, settings, rep);
return rep;
},
K,
std::ref(settings),
&rep);
if (terminating_) {
break;
}
threadpool.emplace_back(std::move(fu));
}
for (auto& fu : threadpool) {
process_finished_rep(fu.get());
}
finished_ = true;
if (!terminating_) {
Logger::Status("\rDone creating geometry (" + boost::lexical_cast(all_processed_elements_.size()) +
" objects) ");
}
}
```
I think this could be somehow related to [https://github.com/emscripten-core/emscripten/issues/8201](this)
ps. I tried setting THREAD_POOL_SIZE to 490 (just as a test) and the iteration goes for more or less 490 objects.
this is an example on how I use the iterator
```
if (geom_iterator_serializer.initialize()) {
do {
// Do my stuff
//Here seems that every iteration uses a new thread/worker
//leading to errors
} while (geom_iterator_serializer.next());
}
```
errors:
```
Tried to spawn a new thread, but the thread pool is exhausted.
This might result in a deadlock unless some threads eventually exit or the code explicitly breaks out to the event loop.
If you want to increase the pool size, use setting `-sPTHREAD_POOL_SIZE=...`.
worker.js onmessage() captured an uncaught exception: std::__2::system_error: thread constructor failed: Resource temporarily unavailable
Uncaught exception from main loop: ffbd51c6-1c0a-44cc-b833-e08ba3ca66c0:9:99729
unwind
Halting program
Pthread 0x4c147998 sent an error! blob:http://localhost:58010/ffbd51c6-1c0a-44cc-b833-e08ba3ca66c0:9: Error: thread constructor failed: Resource temporarily unavailable
```
any tip will be much appreciated!!
Contributor guide
Assessment
This issue has not been assessed yet.