lablup / lablup/backend.ai

Wrap unsafe 'fire-and-forget' asyncio use in RPC calls with bgtask framework

Open
#6,270 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Motivation

- Currently, there are some RPC calls (notably `purge_containers()` and `drop_kernel_registry()`) that call `asyncio.create_task(...)` to run long-running functions and return RPC calls immediately.
- This is unsafe, as we do not properly manage the cancellation and/or failure of such tasks, especially during shutdown cleanup.
- There is a bgtask framework implementation which wraps `TaskContext` and adds progress tracking. We should replace the use of bare task creations with using this framework, or at least the provided `TaskContext` to properly manage failure conditions and cleanup on shutdown.
- If possible, we should also make `_debug_server_task()`, which essentially surrounds server execution with try-except to debug-log any errors raised by the server execution.

## Required Features

- Remove all calls to `asyncio.create_task()` and replace it by using the bgtask framework, or at least `TaskContext`

## Impact

- The way how the affected RPC calls are run might change after this change, but the functionality should not change, and in fact it should be able to better handle cancellations or failures.

## Testing Scenarios

- Add unit tests to mock/fake scenarios where the RPC functions are cancelled or failed.
- Also test what happens when RPC calls are still running when the server begins to shut down.

JIRA Issue: BA-2706

Contributor guide

Open the contributing guide

Research direction

Search the RPC implementations for purge_containers(), drop_kernel_registry(), _debug_server_task(), and all asyncio.create_task() calls; then read the existing bgtask framework and TaskContext lifecycle. Add cancellation, failure, and shutdown coverage for affected RPC calls, and confirm no bare task creation remains while normal functionality is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, distributed-systems
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.