lablup / lablup/backend.ai

Limit unbounded asyncio.gather concurrency in manager to prevent thread/memory leak

Open
#10,735 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Root cause fix for the manager thread/memory leak reported in BA-4210. Multiple unbounded asyncio.gather patterns cause task storms under network partition, DB failover, or Valkey connectivity issues.

## Background

When the manager runs for extended periods, threads and memory grow continuously, eventually leading to OOM restarts (BA-4210). Analysis identified unbounded asyncio.gather as the core cause — under adverse conditions (network partition, DB recovery), hundreds to thousands of concurrent tasks are spawned without any concurrency control.

## Unresolved Patterns (to be split into sub-issues)

1. Agent Lost Event Storm

- Location: agent_lost_checker.py — fires unbounded AgentTerminatedEvents in a loop when multiple agents exceed heartbeat timeout

- Fix: rate-limiting or batching

2. Session/Kernel Sweeper Burst

- Location: registry.py:1359, 1719, 1753 — asyncio.gather(\*rpc_coros) with unbounded concurrent RPCs

- Fix: semaphore to limit concurrent destroy operations

3. Scheduler Bulk Operations

- Location: sokovan/scheduler/terminator/terminator.py, launcher/launcher.py — unbounded gather for terminate/start tasks

- Fix: batching + concurrency limit

4. Contributing Factors (registry.py per_agent_tasks, deployment/executor.py registration_tasks, user/service.py purge tasks)

- All use asyncio.gather with unbounded concurrency

- Fix: apply semaphore to each location

## Already Resolved or Partially Resolved

- Idle Checker: sessions are now processed sequentially (partially resolved)

- Valkey Reconnection: refactored to MonitoringValkeyClient with single monitor task + proper cleanup (mostly resolved, exponential backoff not yet applied)

## Approach

- Use asyncio.Semaphore to cap concurrency

- Batch large operations into chunks

- Introduce coalescing/rate-limiting for event firing

JIRA Issue: BA-5561

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the asyncio.gather call sites named in agent_lost_checker.py, registry.py, sokovan/scheduler/terminator/terminator.py, launcher/launcher.py, deployment/executor.py, and user/service.py. Define separate bounded-concurrency or coalescing changes for the unresolved patterns, then verify that the manager no longer creates unbounded task bursts during agent, session, scheduler, registration, or purge operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.