Limit unbounded asyncio.gather concurrency in manager to prevent thread/memory leak
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Root cause fix for the manager thread/memory leak reported in BA-4210. Multiple unbounded asyncio.gather patterns cause task storms under network partition, DB failover, or Valkey connectivity issues.
## Background
When the manager runs for extended periods, threads and memory grow continuously, eventually leading to OOM restarts (BA-4210). Analysis identified unbounded asyncio.gather as the core cause — under adverse conditions (network partition, DB recovery), hundreds to thousands of concurrent tasks are spawned without any concurrency control.
## Unresolved Patterns (to be split into sub-issues)
1. Agent Lost Event Storm
- Location: agent_lost_checker.py — fires unbounded AgentTerminatedEvents in a loop when multiple agents exceed heartbeat timeout
- Fix: rate-limiting or batching
2. Session/Kernel Sweeper Burst
- Location: registry.py:1359, 1719, 1753 — asyncio.gather(\*rpc_coros) with unbounded concurrent RPCs
- Fix: semaphore to limit concurrent destroy operations
3. Scheduler Bulk Operations
- Location: sokovan/scheduler/terminator/terminator.py, launcher/launcher.py — unbounded gather for terminate/start tasks
- Fix: batching + concurrency limit
4. Contributing Factors (registry.py per_agent_tasks, deployment/executor.py registration_tasks, user/service.py purge tasks)
- All use asyncio.gather with unbounded concurrency
- Fix: apply semaphore to each location
## Already Resolved or Partially Resolved
- Idle Checker: sessions are now processed sequentially (partially resolved)
- Valkey Reconnection: refactored to MonitoringValkeyClient with single monitor task + proper cleanup (mostly resolved, exponential backoff not yet applied)
## Approach
- Use asyncio.Semaphore to cap concurrency
- Batch large operations into chunks
- Introduce coalescing/rate-limiting for event firing
JIRA Issue: BA-5561
Contributor guide
Research direction
Start by reviewing the asyncio.gather call sites named in agent_lost_checker.py, registry.py, sokovan/scheduler/terminator/terminator.py, launcher/launcher.py, deployment/executor.py, and user/service.py. Define separate bounded-concurrency or coalescing changes for the unresolved patterns, then verify that the manager no longer creates unbounded task bursts during agent, session, scheduler, registration, or purge operations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100