lablup / lablup/backend.ai-jail

Child count gets too much increased due to missing exit tracking

Open
#6 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
8
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Often TensorFlow codes spawn many threads, but the jail recognizes "too many" threads while the actual number of threads are within the configured limit.

Potential solutions:
* Directly read "/proc/{pid}/status" to get the actual number of threads from the OS. May incur some overheads when spawning new processes/threads in the child.
* Guard the childCount variable with explicit locks.

But still, TensorFlow seems to increase the number of threads when we repeat calling regressors.
We need to find some good solution on this.

NOTE:
Even the following code produces a large number of threads more than the number of CPU cores allocated to the container:
```
config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1, \
allow_soft_placement=True, device_count = {'CPU': 1})
session = tf.Session(config=config)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the childCount updates and exit tracking in the jail, then reproduce the TensorFlow snippet from the issue while observing the process thread count. Compare the configured limit with the OS-reported count and determine a reliable approach that avoids false "too many" thread detections.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
operating-systems, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.