lablup / lablup/backend.ai

Agent crashes with "Too many open files" due to FD leak in periodic stat collection

Open
#8,806 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Summary

The Backend.AI Agent process exhausts file descriptors over time, resulting in `OSError(24, 'Too many open files')` errors. This causes Docker stats collection to fail for all containers and the commit status reporting task to error out.

## Root Cause

Two periodic tasks accumulate leaked file descriptors:

1. `_report_all_kernel_commit_status_map()` (every 7s, `agent.py:1209-1238`): `Path.iterdir()` and `Path.glob()` generators are not explicitly closed on exception, leaking 1-2 FDs per invocation.
1. **Docker stats collection** (every 5s, `docker/intrinsic.py`): Each container stat query creates a new `Docker()` instance (with its own `aiohttp.ClientSession`), causing 200-300+ FDs to be opened simultaneously for large deployments.

## Affected Code Paths

- `ai.backend.agent.agent:_report_all_kernel_commit_status_map` (agent.py:1209-1238)
- `ai.backend.agent.docker.intrinsic:CPUPlugin.gather_container_measures` (intrinsic.py:275-286)
- `ai.backend.agent.docker.intrinsic:MemPlugin.gather_container_measures` (intrinsic.py:727-761)
- `ai.backend.agent.stats:collect_container_stat` (stats.py:745-750)

JIRA Issue: BA-4389

Contributor guide

Open the contributing guide

Research direction

Start with _report_all_kernel_commit_status_map in agent.py:1209-1238, then inspect CPUPlugin.gather_container_measures and MemPlugin.gather_container_measures in docker/intrinsic.py alongside collect_container_stat in stats.py:745-750. Trace how periodic calls handle Path iterators and Docker instances; done means repeated status and container-stat collection no longer accumulates file descriptors or causes OSError(24).

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.