chaoss / chaoss/CollectOSS

shutdown Celery workers do not have an opportunity to reset their task status

Open
#388 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13
Forks
17
Avg merge
6h 59m
Merged PRs (30d)
1

Description

We observed in production that Flower was showing 0 core tasks running for an extended period of time, yet the `collection_status` operations table had ~40 core tasks listed as `Collecting` (with UUIDs).

confirming (by searching the repo url for those repos in flower) that these tasks were not currently running and hadnt already completed, we concluded that they were stale/the DB's picture of collection status had gotten out of sync with flower.

Changing a few of these tasks from `Collecting` with a UUID, to `Pending` (and nulling out the UUID) seems to have partially cleared the block on core workers.

When asked to identify bugs that could cause this out of sync-ness/stale tasks, Generative AI identified 4 issues. Three of these were deemed (by me) to be not relevant and/or intended behavior, but the one that was promising seemed to be:

> No on_revoke / task revocation handler
>
> celery_app.py Lines 1-10
> from celery.signals import worker_process_init, worker_process_shutdown
> There's no task_revoked or task_prerun/task_postrun signal handler that would reset COLLECTING to ERROR/PENDING when tasks are revoked during worker shutdown. The on_failure on CoreRepoCollectionTask is only triggered by exceptions raised inside the task function — not by SIGKILL, OOM kills, or Celery task revocation.

This seems like a very plausible error chain:
1. a celery worker gets shutdown or recreated for some reason
2. we have no error handler to catch this when that workers tasks are revoked
3. we subsequently never change the status of those now-revoked tasks in the DB
4. even if/when the new workers come back, the collection monitor doesnt pick up any more tasks because it thinks core collection is already running at full beans

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.