Issue running rstan with BiocParallel MulticoreParam

Summary:

When running the same stan model in parallel on a high number of datasets on multiple cores, using BiocParallel package with MulticoreParam back-end, I get an error message linked to shared objects loading.

Description:

In the context of a proteomics research work, I’d like to run the same model on a number of different datasets (1000+) in parallel on a number of cores (6 in my case). For this I am trying to use BiocParallel with its MulticoreParam back-end (on Linux). Note I don’t use the paralelization feature present in Stan (cores = 1).

This works fine for a fairly reasonable number of models (100) but when increasing further the number of models, while keeping the same number of cores, I get systematically an error message :

Error: BiocParallel errors
element index: 271 (or other element index depending on run)
unable to load shared object ‘tmp/Rtmp7TxrWl/file249d104be97c44.so’
tmp/Rtmp7TxrWl/file249d104be97c44.so : file too short

and as soon as this happens the rest of the jobs all fail with the same
type of error message.

I tried to run the batch in serial mode (SerialParam in BiocParallel) and this works fine, so it is unlikely to be due to the data specifics of one model in the series.

Since I suspected it might be related to a resource shortage issue (e.g. memory), I also tried to decrease the number of cores used in order to limit the number of jobs run simultaneously, but even with 2 cores the issue appears. I also tried to decrease the number of chain iterations to a very low number but again the issue is still there.

Anyone having experienced the same kind of issue in the past and having found a solution for this ?

Reproducible Steps:

Difficult to provide anything reproducible, since I understand that to get the error you should have the same environment (OS etc.)

RStan Version:

2.21.2

R Version:

4.0.3

Operating System:

Manjaro Linux 20.2.1

Start by reproducing the workload described with rstan 2.21.2, R 4.0.3, Manjaro Linux, and BiocParallel's MulticoreParam, then compare it with SerialParam. Focus on the shared-object loading error that appears after many datasets and record whether the failure can be reproduced with fewer cores or iterations. Done requires a reproducible cause and a documented resolution.