docling-project / docling-project/docling-jobkit

fix(rq): watchdog cleanup calls abstract StartedJobRegistry.remove

Open Beginner friendly
#237 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36
Forks
47
Avg merge
4d 22h
Merged PRs (30d)
15

Description

## AI disclosure

I encountered this issue in my deployment. This issue report was prepared with assistance from AI.

## Description

The RQ watchdog cleanup path introduced for #106 calls `StartedJobRegistry.remove(task_id)`. In current RQ 2.x, `StartedJobRegistry.remove()` is intentionally unimplemented and raises `NotImplementedError`.

With `docling-jobkit==3.4.0` and `rq==2.11.0`, an orphaned worker is detected and the task is published as `FAILURE`, but cleanup then fails:

```text
Failed to clean up RQ state for task ...
NotImplementedError
```

The job remains in `rq:wip:{queue}`, so the watchdog can rediscover it and emit repeated warnings.

## Deterministic reproduction

The underlying RQ failure can be reproduced without Redis:

```bash
uv run python -c "from rq.registry import StartedJobRegistry; StartedJobRegistry(name='convert', connection=None).remove('repro-task')"
```

This raises `NotImplementedError` because `StartedJobRegistry.remove()` is abstract in RQ 2.x.

## Watchdog reproduction

1. Run `docling-jobkit 3.4.0` with RQ 2.x.
2. Let a `SimpleWorker` job lose its heartbeat, or reproduce the watchdog branch with a missing heartbeat key and a started-job registry entry.
3. Wait for `_watchdog_task()` to process the orphaned task.
4. Observe the `StartedJobRegistry.remove()` `NotImplementedError` in the cleanup traceback.

## Expected behavior

The watchdog should set the RQ job to `FAILED` and remove its started execution entry without raising. Missing or already-expired jobs should remain a non-fatal cleanup case.

## Proposed fix

Use the RQ execution-aware cleanup API, such as `StartedJobRegistry.remove_executions(job)`, after fetching the job, and add a regression test covering the cleanup path. This is a follow-up to the implementation released for #106.

## Environment

- docling-jobkit 3.4.0
- rq 2.11.0
- Python 3.12 or 3.13

Contributor guide

Open the contributing guide

Research direction

Start at the watchdog entry point `_watchdog_task()` and reproduce the underlying RQ failure with the provided `uv run python` command. Trace the orphan cleanup path and add a regression test for it; done means the job is marked `FAILED`, its started execution entry is removed, and missing or expired jobs do not raise.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.