Agent crashes with "Too many open files" due to FD leak in periodic stat collection
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Summary
The Backend.AI Agent process exhausts file descriptors over time, resulting in `OSError(24, 'Too many open files')` errors. This causes Docker stats collection to fail for all containers and the commit status reporting task to error out.
## Root Cause
Two periodic tasks accumulate leaked file descriptors:
1. `_report_all_kernel_commit_status_map()` (every 7s, `agent.py:1209-1238`): `Path.iterdir()` and `Path.glob()` generators are not explicitly closed on exception, leaking 1-2 FDs per invocation.
1. **Docker stats collection** (every 5s, `docker/intrinsic.py`): Each container stat query creates a new `Docker()` instance (with its own `aiohttp.ClientSession`), causing 200-300+ FDs to be opened simultaneously for large deployments.
## Affected Code Paths
- `ai.backend.agent.agent:_report_all_kernel_commit_status_map` (agent.py:1209-1238)
- `ai.backend.agent.docker.intrinsic:CPUPlugin.gather_container_measures` (intrinsic.py:275-286)
- `ai.backend.agent.docker.intrinsic:MemPlugin.gather_container_measures` (intrinsic.py:727-761)
- `ai.backend.agent.stats:collect_container_stat` (stats.py:745-750)
JIRA Issue: BA-4389
Contributor guide
Research direction
Start with _report_all_kernel_commit_status_map in agent.py:1209-1238, then inspect CPUPlugin.gather_container_measures and MemPlugin.gather_container_measures in docker/intrinsic.py alongside collect_container_stat in stats.py:745-750. Trace how periodic calls handle Path iterators and Docker instances; done means repeated status and container-stat collection no longer accumulates file descriptors or causes OSError(24).
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100