Deleting a knowledge base while its documents are still indexing leaves orphaned segments, child_chunks and pgvector tables

### Self Checks

- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report.
- [x] Please do not modify this template :) and fill in all the required fields.

### Dify version

1.x (self-hosted, Community Edition), pgvector vector store, parent-child chunker

### Cloud or Self Hosted

Self Hosted (Docker)

### Steps to reproduce

1. Create a Knowledge base backed by a RAG pipeline and start ingesting a **large** source (tens of thousands of documents), so indexing runs for a while.
2. While the documents are **still indexing**, delete that Knowledge base from the UI (`DELETE /console/api/datasets/{id}`).

### ✔️ Expected Behavior

Deleting a dataset should first **cancel/await the in-flight indexing** for that dataset, then clean up completely, leaving **no orphaned rows or vector tables**: `documents`, `document_segments`, `child_chunks`, dataset_* rows and the `embedding_vector_index__nod` pgvector table should all be gone.

### ❌ Actual Behavior

The dataset row is removed (`DELETE` returns 204), but the cleanup does **not** complete and large amounts of data are **orphaned**:

- The still-running indexing pipeline keeps executing against the deleted dataset and repeatedly fails with `KnowledgeIndexNodeError: Dataset not found.` (`core/rag/index_processor/index_processor.py` `index_and_clean`), producing a storm of `Node knowledgeBase failed with ABORT strategy` errors.
- `clean_dataset_task` does not effectively remove the data. In our case, after deleting 3 knowledge bases mid-indexing we were left with:
- `document_segments`: ~2.4M orphaned rows (dataset_id no longer in `datasets`)
- `child_chunks`: ~830k orphaned rows
- 3 orphaned `embedding_vector_index__nod` pgvector tables (~34 GB)
- `documents` / `dataset_process_rules`: ~27k / ~25k orphaned rows

These were never cleaned and had to be removed manually.

### Root-cause notes

Two compounding problems:

1. **No guard against deleting a dataset that is actively indexing.** `DatasetService.delete_dataset` (`services/dataset_service.py`) deletes the dataset and dispatches `clean_dataset_task` via the `clean_when_dataset_deleted` event, but does not cancel or await in-flight indexing. The concurrent `knowledge_index` node keeps writing/reading for the now-deleted dataset (race), and its long-lived transactions can block the cleanup deletes.

2. **`clean_dataset_task` is not robust for large datasets.** `tasks/clean_dataset_task.py` loads *all* segments into memory and deletes them in a single transaction using large `IN (...)` statements. On a multi-million-row dataset this can time out or be blocked by the indexing transactions; any exception rolls back the **entire** task (`session.rollback()`), so nothing is deleted and everything is left orphaned. The vector-store `clean()` is in its own try/except, but if it also fails/rolls back, the `embedding_vector_index_*` table survives too.

### Suggested fix

- On dataset delete, first stop/cancel active indexing for that dataset (or refuse deletion while indexing, returning a clear error) before removing the dataset and dispatching cleanup.
- Make `clean_dataset_task` delete in **bounded batches with per-batch commits** so a timeout doesn't roll back the whole cleanup, and make vector-table cleanup idempotent/retryable.

### Related issues

- #9354 — deleting a document does not clean up its index results
- #24407 — race condition in parent/child segment deletion leaving orphaned child chunks
- #12041 — Internal Server Error when modifying/deleting a knowledge base
- #13194 — Document stuck at Queuing

Trace dataset deletion through services/dataset_service.py and the clean_when_dataset_deleted event, then inspect tasks/clean_dataset_task.py and core/rag/index_processor/index_processor.py. Reproduce deletion during active indexing with the pgvector setup and observe cleanup behavior. Done means indexing no longer races with deletion and documents, segments, child_chunks, dataset_process_rules, and embedding_vector_index tables are fully removed.

Written by the indexing model from the issue text.