Deleting a dataset leaves an orphaned vector collection when `doc_form` or `indexing_technique` is empty (vector DB never cleaned up)
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
### Dify version
v1.15.0
### Cloud or Self Hosted
Self Hosted (Docker)
### Steps to reproduce
1. Deploy Dify with `VECTOR_STORE=qdrant`.
2. Use a dataset whose `datasets.doc_form` and/or `datasets.indexing_technique` column is empty/null. This is common for knowledge bases created in older Dify versions and carried forward via upgrade. Index some documents so a Qdrant collection such as `Vector_index__Node` exists on the vector store.
3. Delete the dataset from the UI (or via `DELETE /datasets/{id}`).
4. Inspect Qdrant afterwards:
```
GET /collections
```
The `Vector_index__Node` collection is **still present**. Checking the Qdrant HTTP access log shows that **no `DELETE /collections/...` request was ever issued** during the deletion.
5. The orphaned collection persists permanently. Repeated deletes keep accumulating more orphans; they are never reconciled.
### ✔️ Expected Behavior
When a dataset is deleted, the vector-store collection that holds its embeddings must be reliably deleted (or scheduled for deletion that is guaranteed to run), so that no orphaned collection is left behind — regardless of whether `doc_form` / `indexing_technique` are populated.
### ❌ Actual Behavior
The `datasets` row is removed, but the corresponding Qdrant collection is **never deleted** and becomes an orphan. The deletion handler short-circuits before dispatching the cleanup task, and there is no retry/reconcile path, so the orphan is permanent.
### Root cause (code-level)
The signal handler that owns vector cleanup guards too aggressively and bails out for exactly the case the cleanup task is prepared to handle.
`api/events/event_handlers/clean_when_dataset_deleted.py`:
```python
@dataset_was_deleted.connect
def handle(sender: Dataset, **kwargs):
dataset = sender
if not dataset.doc_form or not dataset.indexing_technique:
return # ← skips vector cleanup entirely
clean_dataset_task.delay(
dataset.id,
dataset.tenant_id,
dataset.indexing_technique,
dataset.index_struct,
dataset.collection_binding_id,
dataset.doc_form,
dataset.pipeline_id,
)
```
The early `return` short-circuits cleanup for any dataset whose `doc_form` **or** `indexing_technique` is empty. Datasets created in older Dify versions (and migrated forward) frequently have one of these fields empty, so deleting them never dispatches `clean_dataset_task`, and the Qdrant collection is never removed.
This is also internally inconsistent. The very task it guards already defends against an empty `doc_form`:
```python
# api/tasks/clean_dataset_task.py
if doc_form is None or (isinstance(doc_form, str) and not doc_form.strip()):
# Use default paragraph index type for empty/invalid datasets
# to enable vector database cleanup
doc_form = IndexStructureType.PARAGRAPH_INDEX
```
So the handler's guard is **stricter than the task it protects** — it blocks the precise case the task already handles. `indexing_technique` is likewise not required by the cleanup logic (it is only stored on the reconstructed `Dataset` object and not used to select the cleanup path), so guarding on it serves no purpose other than skipping cleanup.
A second, related weakness compounds the problem. Inside `clean_dataset_task`, the vector cleanup is wrapped in `try/except`; on failure it only logs and then proceeds to commit the rest of the DB cleanup:
```python
# api/tasks/clean_dataset_task.py
try:
index_processor = IndexProcessorFactory(doc_form).init_index_processor()
index_processor.clean(dataset, None, with_keywords=True, delete_child_chunks=True)
except Exception:
logger.exception(...)
# Continue with document and segment deletion even if vector cleanup fails
```
Because `delete_dataset` has already committed the deletion of the `datasets` row (`db.session.delete(dataset)` + `commit`) before the async task runs, a failed cleanup can never be retried — the orphan is permanent and silent.
### Impact
Orphans accumulate silently and invisibly (the API never reports them). On a qdrant container restart/recreate, **all** collections — including orphans — are loaded at once, spiking open file handles. If the container's `nofile` ulimit is modest, this triggers RocksDB `IO error: Too many open files` → Qdrant panic → `restart: always` loop → all knowledge-base retrieval fails. From the API side this surfaces as:
```
InactiveRpcError ... StatusCode.UNAVAILABLE
details: "DNS resolution failed for qdrant:6334: ... Domain name not found"
```
(The DNS error is a symptom: the qdrant container is crash-looping and intermittently unresolvable, not an actual DNS problem.)
We hit exactly this in production after a version upgrade that recreated the qdrant container: the container restart-looped ~100 times and knowledge-base retrieval was down.
Contributor guide
Research direction
Start with api/events/event_handlers/clean_when_dataset_deleted.py and api/tasks/clean_dataset_task.py, then reproduce deletion with empty doc_form or indexing_technique values against Qdrant. Trace whether cleanup is dispatched and whether the vector collection is removed; done means these datasets no longer leave orphaned collections and the existing failure behavior is addressed or explicitly covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100