alan-turing-institute / alan-turing-institute/PosteriorBootstrap

Unable to run long parallel code on more than 18 cores on Azure

Open
#65 0 comments 0 reactions 0 assignees View on GitHub
on hold
Dominant language
R
Stars
4
Forks
3
Avg merge
5h 18m
Merged PRs (30d)
1

Description

The code for assessing the speedup is failing on Azure. It succeeds from 1 to 64 cores with `n = 1,000`and gave this error twice at the same spot with `n = 10,000`:

```
[1] "n_bootstrap = 10000"
[1] "num_cores = 1"
...
[1] "num_cores = 17"
[1] "Finished sampling"
[1] "num_cores = 18"
Error in mcfork() :
unable to fork, possible reason: Resource temporarily unavailable
Calls: source ... -> -> mclapply -> lapply -> FUN -> mcfork
Execution halted
Error while shutting down parallel: unable to terminate some child processes
```

Restarting the VM didn't help.

The speed-up ran successfully on 4th July 2019 and the code is exactly the same as today. I will not attempt to debug this problem.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the speedup code that calls R's mclapply and reproduce the failure on Azure using n = 1,000 and n = 10,000 across increasing core counts. Inspect the mcfork resource error and child-process shutdown behavior. Done means the speedup run completes reliably beyond 18 cores, or the issue documents a confirmed platform limitation and safe workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, r
Domain
cloud, hpc, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.