ethereum / ethereum/async-service

Race condition causing Lifecycle error during cancellation

Open
#72 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
11
PR merge metrics
No merged PRs in 30d

Description

_If this is a bug report, please fill in the following sections.
If this is a feature request, delete and describe what you would like with examples._

## What was wrong?

In the trinity codebase this warning happens occasionally

```sh
: unexpected error when cancelling tasks, service may not terminate
Traceback (most recent call last):
File "/home/piper/python-environments/trinity/lib/python3.8/site-packages/async_service/asyncio.py", line 122, in _handle_cancelled
await self._real_handle_cancelled()
File "/home/piper/python-environments/trinity/lib/python3.8/site-packages/async_service/asyncio.py", line 170, in _real_handle_cancelled
raise LifecycleError(f"Should have already been completed: {asyncio_task}")
async_service.exceptions.LifecycleError: Should have already been completed: wait_for=()]>>

```

This *appears* to be the result of a new task being scheduled while cancellation is in progress.

## How can it be fixed?

Look into how to reproduce this and then figure out how to prevent it? This could be as simple as a small wait to allow any tasks that are in the process of being scheduled to land into the data structures before proceeding with task cancellation, but ideally we can land on a mechanism that is more precise.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by investigating _handle_cancelled and _real_handle_cancelled in async_service/asyncio.py, then trace BaseManager._run_and_manage_task at async_service/base.py:300. Reproduce cancellation while a new task is being scheduled and determine how the race occurs. Done means the cancellation path no longer raises LifecycleError when task scheduling overlaps cancellation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.