langgenius / langgenius/dify

Annotation reply jobs can overlap per app and remain stuck in waiting

Open
#40,595 0 comments 1 reaction 0 assignees View on GitHub
🐞 bug
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.

### Dify version

main (7aba539e82, 2026-08-12)

### Cloud or Self Hosted

Self Hosted (Source)

### Steps to reproduce

1. Run the API and Celery worker from current `main`.
2. For the same app, call `AppAnnotationService.enable_app_annotation(...)` twice before the first task finishes.
3. Observe that both requests receive different job IDs and both enqueue `enable_annotation_reply_task`.
4. While an enable task is in progress, call `disable_app_annotation(...)` for the same app. Enable and disable use separate app-level keys, so the two operations are not mutually exclusive.
5. Let a queued task run after the app was deleted, or disable annotation reply when its setting is already absent. These early-return paths do not write a terminal job status.

The service checks `enable_app_annotation_{app_id}` and `disable_app_annotation_{app_id}`, but it never writes either key when enqueueing a task. It only creates the per-job status key with `setnx`, without an expiration. The tasks delete the app-level keys even though those keys were never acquired.

Related: #37646 reported the missing app-level key write and was closed as stale after its proposed fixes were not merged. The same code path is still present on current `main`; the shared enable/disable exclusion and terminal-status paths are also affected.

### ✔️ Expected Behavior

Annotation reply mutations should have one application-scoped job lifecycle:

- only one enable or disable job may mutate an app at a time;
- repeated requests for the same action should reuse the in-flight job;
- a conflicting action should not start concurrently;
- enqueue failures and all worker exit paths should release the application reservation;
- every accepted job should transition from waiting to processing and then to completed or error.

### ❌ Actual Behavior

Multiple jobs can be queued for one app and can concurrently rebuild/delete the same annotation vector data while updating or deleting the same annotation setting. Some early-return paths leave the job status at waiting indefinitely, so the caller can continue polling a job that will never finish.

Contributor guide

Open the contributing guide

Research direction

Start with AppAnnotationService.enable_app_annotation(...) and disable_app_annotation(...), then trace enable_annotation_reply_task and the application and per-job status keys. Reproduce repeated and conflicting requests with the API and Celery worker. Done means one application-scoped lifecycle prevents overlap, accepted jobs reach a terminal status, and enqueue or worker exit paths release reservations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.