aims-umich / aims-umich/neorl

-bash: fork: retry: Resource temporarily unavailable...NEORL starting many sleeping processes

Ouverte
#16 1 commentaire 1 réaction 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
71
Forks
18
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

This post is for users who may encounter:
`-bash: fork: retry: Resource temporarily unavailable`
or
`Segmentation fault (core dumped) nohup python de_expl.py`
errors.

On linux computers, users often have a maximum number of processes they are allowed to have running on a computer. This can be checked with the `ulimit -a` command under the `max user processes` row. For example:
```
>> ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1028858
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
```

A user can check how many processes (in total) they have running with the command: `ps --no-headers auxwwwm | awk '$2 == "-" { print $1 }' | sort | uniq -c | sort -n`. If a user wants to see the specific listing: `ps --no-headers auxwwwm`.

A typical Python program may start <50 processes. I am not exactly sure why this is the case but I checked a few different random scripts I had lying around and this is the conclusion I came to.

For some reason, when running NEORL in serial, around 300 processes are started. I think this has something to do with parallelization. Most of the processes are sleeping, for whatever reason.

This becomes a problem if a user wants to run multiple independent python programs which use NEORL. Regardless of computer size, the process limit is quickly reached. A fix to this is to simply raise the max number of user processes: `ulimit -u ####`. But this cannot be done without `sudo` access.

I do not think this is an urgent problem for NEORL as it only comes up in a specific use case but I wanted to post this to provide information to users who encounter the same problem.

Possibly relevant links:
https://stackoverflow.com/questions/20614309/find-reason-for-sleeping-python-process
https://stackoverflow.com/questions/31193449/python-multiprocessing-big-data-turn-process-into-sleep
https://stackoverflow.com/questions/1032813/dump-stacktraces-of-all-active-threads/7317379#7317379

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.