Fix persistent stat collection failures caused by stale container resources
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Agent stat collection (container/node/process) permanently hangs due to stale container network namespaces.
Symptoms:
- "unhandled exception while syncing container stats" (TimeoutError) repeating
- "unhandled exception while syncing node stats" (TimeoutError) co-occurring
- "unhandled exception while syncing process stats" (TimeoutError) co-occurring
- CPUPlugin/MemoryPlugin: sysfs unreadable warnings followed by netstat_ns hang
- Once triggered, repeats indefinitely until agent restart
Root causes:
1. MemoryPlugin sysfs_impl calls container.show() and netstat_ns() without per-call timeouts — entering a dead namespace hangs the thread indefinitely
2. netstat_ns_work runs in thread executor (run_in_executor) and cannot be cancelled by asyncio — hung threads accumulate and exhaust the default thread pool
3. stats.py plugin gather calls lack per-plugin timeouts — a single hung plugin blocks all stat results until STAT_COLLECTION_TIMEOUT (10 min)
4. Deployed version (bai-25.14) uses a shared self._lock across collect_node_stat / collect_container_stat / collect_per_container_process_stat — container stat hang propagates to node and process stat collection
Fix scope:
- Add per-call timeouts to container.show() and netstat_ns() — BA-4933
- Add per-plugin timeouts to stats.py gather calls — BA-4934
- Pre-validate namespace path or check container state before netstat_ns to prevent thread leaks
- Confirm shared lock removal in main is deployed (already removed in main, needs release)
JIRA Issue: BA-4937
Contributor guide
Assessment
This issue has not been assessed yet.