pulp / pulp/pulpcore

Immediate task timeout leaves task stuck in RUNNING state

Open
#7,599 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Issue
Dominant language
Python
Stars
598
Forks
168
Avg merge
1d 4h
Merged PRs (30d)
86

Description

Summary

When an immediate task exceeds the 5-second IMMEDIATE_TIMEOUT, the log message is
correctly produced:

pulpcore.tasking.tasks:INFO: Immediate task <uuid> timed out after 5 seconds.

However, the task can remain stuck in running state rather than transitioning to
failed. This affects both the PostgreSQL and Redis (WORKER_TYPE=redis) worker paths,
since both share the same _execute_task/_aexecute_task code path.

Root cause

When asyncio.wait_for cancels the inner coroutine on timeout, Django's
sync_to_async (with thread_sensitive=True) serializes all ORM operations through
the main thread. The cancelled coroutine's thread may still be running a database
operation, blocking the main thread queue. The subsequent set_failed call (which
needs the same main thread) is therefore delayed. During this window the task appears
stuck in running.

If a cancel request arrives during this delay:

  • set_canceling() and the delayed set_failed race to UPDATE the task row.
  • If set_failed wins: task transitions to failed with the timeout message.
  • If set_canceling wins: task transitions to canceling; set_failed then finds
    0 matching rows (state is no longer running) and raises RuntimeError, which
    propagates up uncaught, leaving the task stuck in canceling until a worker
    eventually cleans it up as canceled.

Expected behavior

The task transitions to failed immediately after the 5-second timeout with an error
describing the timeout.

Actual behavior

The task remains in running state. It may eventually transition to failed (with the
timeout error) after a delay, or to canceled if a cancel request races ahead of the
delayed set_failed.

Environment

  • pulpcore version: 3.108.0
  • WORKER_TYPE: redis

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the shared _execute_task and _aexecute_task paths, focusing on the asyncio.wait_for timeout and the subsequent set_failed call. Reproduce an immediate task exceeding IMMEDIATE_TIMEOUT through both PostgreSQL and Redis worker paths, including a concurrent cancel request. Done means the task reaches failed promptly with the timeout error and does not remain stuck in running or canceling.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, postgresql, python, redis
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.