lablup / lablup/backend.ai

Fix persistent stat collection failures caused by stale container resources

Open
#9,778 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Agent stat collection (container/node/process) permanently hangs due to stale container network namespaces.

Symptoms:

- "unhandled exception while syncing container stats" (TimeoutError) repeating

- "unhandled exception while syncing node stats" (TimeoutError) co-occurring

- "unhandled exception while syncing process stats" (TimeoutError) co-occurring

- CPUPlugin/MemoryPlugin: sysfs unreadable warnings followed by netstat_ns hang

- Once triggered, repeats indefinitely until agent restart

Root causes:

1. MemoryPlugin sysfs_impl calls container.show() and netstat_ns() without per-call timeouts — entering a dead namespace hangs the thread indefinitely

2. netstat_ns_work runs in thread executor (run_in_executor) and cannot be cancelled by asyncio — hung threads accumulate and exhaust the default thread pool

3. stats.py plugin gather calls lack per-plugin timeouts — a single hung plugin blocks all stat results until STAT_COLLECTION_TIMEOUT (10 min)

4. Deployed version (bai-25.14) uses a shared self._lock across collect_node_stat / collect_container_stat / collect_per_container_process_stat — container stat hang propagates to node and process stat collection

Fix scope:

- Add per-call timeouts to container.show() and netstat_ns() — BA-4933

- Add per-plugin timeouts to stats.py gather calls — BA-4934

- Pre-validate namespace path or check container state before netstat_ns to prevent thread leaks

- Confirm shared lock removal in main is deployed (already removed in main, needs release)

JIRA Issue: BA-4937

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.