lablup / lablup/backend.ai-jail
Child count gets too much increased due to missing exit tracking
- Dominant language
- Rust
- Stars
- 8
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Often TensorFlow codes spawn many threads, but the jail recognizes "too many" threads while the actual number of threads are within the configured limit.
Potential solutions:
* Directly read "/proc/{pid}/status" to get the actual number of threads from the OS. May incur some overheads when spawning new processes/threads in the child.
* Guard the childCount variable with explicit locks.
But still, TensorFlow seems to increase the number of threads when we repeat calling regressors.
We need to find some good solution on this.
NOTE:
Even the following code produces a large number of threads more than the number of CPU cores allocated to the container:
```
config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1, \
allow_soft_placement=True, device_count = {'CPU': 1})
session = tf.Session(config=config)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the childCount updates and exit tracking in the jail, then reproduce the TensorFlow snippet from the issue while observing the process thread count. Compare the configured limit with the OS-reported count and determine a reliable approach that avoids false "too many" thread detections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- operating-systems, security
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100