-bash: fork: retry: Resource temporarily unavailable...NEORL starting many sleeping processes
- Dominant language
- Python
- Stars
- 71
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
This post is for users who may encounter:
`-bash: fork: retry: Resource temporarily unavailable`
or
`Segmentation fault (core dumped) nohup python de_expl.py`
errors.
On linux computers, users often have a maximum number of processes they are allowed to have running on a computer. This can be checked with the `ulimit -a` command under the `max user processes` row. For example:
```
>> ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1028858
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
```
A user can check how many processes (in total) they have running with the command: `ps --no-headers auxwwwm | awk '$2 == "-" { print $1 }' | sort | uniq -c | sort -n`. If a user wants to see the specific listing: `ps --no-headers auxwwwm`.
A typical Python program may start <50 processes. I am not exactly sure why this is the case but I checked a few different random scripts I had lying around and this is the conclusion I came to.
For some reason, when running NEORL in serial, around 300 processes are started. I think this has something to do with parallelization. Most of the processes are sleeping, for whatever reason.
This becomes a problem if a user wants to run multiple independent python programs which use NEORL. Regardless of computer size, the process limit is quickly reached. A fix to this is to simply raise the max number of user processes: `ulimit -u ####`. But this cannot be done without `sudo` access.
I do not think this is an urgent problem for NEORL as it only comes up in a specific use case but I wanted to post this to provide information to users who encounter the same problem.
Possibly relevant links:
https://stackoverflow.com/questions/20614309/find-reason-for-sleeping-python-process
https://stackoverflow.com/questions/31193449/python-multiprocessing-big-data-turn-process-into-sleep
https://stackoverflow.com/questions/1032813/dump-stacktraces-of-all-active-threads/7317379#7317379
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with de_expl.py and the NEORL serial execution path, then use ulimit -a and the supplied ps/awk command to measure process counts. Determine why serial runs create many sleeping processes and define done as preventing unnecessary process creation or clearly documenting the limitation and workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, python
- Domain
- operating-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100