alan-turing-institute / alan-turing-institute/PosteriorBootstrap
Unable to run long parallel code on more than 18 cores on Azure
- Dominant language
- R
- Stars
- 4
- Forks
- 3
- Avg merge
- 5h 18m
- Merged PRs (30d)
- 1
Description
The code for assessing the speedup is failing on Azure. It succeeds from 1 to 64 cores with `n = 1,000`and gave this error twice at the same spot with `n = 10,000`:
```
[1] "n_bootstrap = 10000"
[1] "num_cores = 1"
...
[1] "num_cores = 17"
[1] "Finished sampling"
[1] "num_cores = 18"
Error in mcfork() :
unable to fork, possible reason: Resource temporarily unavailable
Calls: source ... -> -> mclapply -> lapply -> FUN -> mcfork
Execution halted
Error while shutting down parallel: unable to terminate some child processes
```
Restarting the VM didn't help.
The speed-up ran successfully on 4th July 2019 and the code is exactly the same as today. I will not attempt to debug this problem.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the speedup code that calls R's mclapply and reproduce the failure on Azure using n = 1,000 and n = 10,000 across increasing core counts. Inspect the mcfork resource error and child-process shutdown behavior. Done means the speedup run completes reliably beyond 18 cores, or the issue documents a confirmed platform limitation and safe workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, r
- Domain
- cloud, hpc, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100