Annotation reply jobs can overlap per app and remain stuck in waiting
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
### Dify version
main (7aba539e82, 2026-08-12)
### Cloud or Self Hosted
Self Hosted (Source)
### Steps to reproduce
1. Run the API and Celery worker from current `main`.
2. For the same app, call `AppAnnotationService.enable_app_annotation(...)` twice before the first task finishes.
3. Observe that both requests receive different job IDs and both enqueue `enable_annotation_reply_task`.
4. While an enable task is in progress, call `disable_app_annotation(...)` for the same app. Enable and disable use separate app-level keys, so the two operations are not mutually exclusive.
5. Let a queued task run after the app was deleted, or disable annotation reply when its setting is already absent. These early-return paths do not write a terminal job status.
The service checks `enable_app_annotation_{app_id}` and `disable_app_annotation_{app_id}`, but it never writes either key when enqueueing a task. It only creates the per-job status key with `setnx`, without an expiration. The tasks delete the app-level keys even though those keys were never acquired.
Related: #37646 reported the missing app-level key write and was closed as stale after its proposed fixes were not merged. The same code path is still present on current `main`; the shared enable/disable exclusion and terminal-status paths are also affected.
### ✔️ Expected Behavior
Annotation reply mutations should have one application-scoped job lifecycle:
- only one enable or disable job may mutate an app at a time;
- repeated requests for the same action should reuse the in-flight job;
- a conflicting action should not start concurrently;
- enqueue failures and all worker exit paths should release the application reservation;
- every accepted job should transition from waiting to processing and then to completed or error.
### ❌ Actual Behavior
Multiple jobs can be queued for one app and can concurrently rebuild/delete the same annotation vector data while updating or deleting the same annotation setting. Some early-return paths leave the job status at waiting indefinitely, so the caller can continue polling a job that will never finish.
Contributor guide
Research direction
Start with AppAnnotationService.enable_app_annotation(...) and disable_app_annotation(...), then trace enable_annotation_reply_task and the application and per-job status keys. Reproduce repeated and conflicting requests with the API and Celery worker. Done means one application-scoped lifecycle prevents overlap, accepted jobs reach a terminal status, and enqueue or worker exit paths release reservations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100