Immediate task timeout leaves task stuck in RUNNING state
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 598
- Forks
- 168
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 86
Description
Summary
When an immediate task exceeds the 5-second IMMEDIATE_TIMEOUT, the log message is
correctly produced:
pulpcore.tasking.tasks:INFO: Immediate task <uuid> timed out after 5 seconds.
However, the task can remain stuck in running state rather than transitioning to
failed. This affects both the PostgreSQL and Redis (WORKER_TYPE=redis) worker paths,
since both share the same _execute_task/_aexecute_task code path.
Root cause
When asyncio.wait_for cancels the inner coroutine on timeout, Django's
sync_to_async (with thread_sensitive=True) serializes all ORM operations through
the main thread. The cancelled coroutine's thread may still be running a database
operation, blocking the main thread queue. The subsequent set_failed call (which
needs the same main thread) is therefore delayed. During this window the task appears
stuck in running.
If a cancel request arrives during this delay:
set_canceling()and the delayedset_failedrace to UPDATE the task row.- If
set_failedwins: task transitions tofailedwith the timeout message. - If
set_cancelingwins: task transitions tocanceling;set_failedthen finds
0 matching rows (state is no longerrunning) and raisesRuntimeError, which
propagates up uncaught, leaving the task stuck incancelinguntil a worker
eventually cleans it up ascanceled.
Expected behavior
The task transitions to failed immediately after the 5-second timeout with an error
describing the timeout.
Actual behavior
The task remains in running state. It may eventually transition to failed (with the
timeout error) after a delay, or to canceled if a cancel request races ahead of the
delayed set_failed.
Environment
- pulpcore version: 3.108.0
WORKER_TYPE:redis
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the shared _execute_task and _aexecute_task paths, focusing on the asyncio.wait_for timeout and the subsequent set_failed call. Reproduce an immediate task exceeding IMMEDIATE_TIMEOUT through both PostgreSQL and Redis worker paths, including a concurrent cancel request. Done means the task reaches failed promptly with the timeout error and does not remain stuck in running or canceling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- django, postgresql, python, redis
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100