Batch segment import stays stuck at waiting when CSV processing fails
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Describe the bug
When importing segments from a CSV, only failures during the initial setup phase mark the
job as failed. If the CSV parsing or the indexing work fails afterwards, the Redis job
status is left at `waiting` (which is written without a TTL), so the status endpoint keeps
returning `waiting` and the frontend keeps polling indefinitely. The user sees an import
that never finishes and never reports an error.
### Steps to reproduce
1. Create a knowledge base document and open "Batch add segments".
2. Upload a CSV that fails during processing, for example:
- an empty CSV (header only, no rows), or
- a QA dataset with a CSV that has only one column (no answer column).
3. Watch the batch import status.
### Expected behavior
The job status becomes `error`, and the UI stops polling and shows the failure.
### Actual behavior
The job status stays at `waiting` forever. The worker logs a traceback, but the UI never
learns about it and keeps polling every few seconds.
### Root cause
In `api/tasks/batch_create_segment_to_index_task.py`, only the setup `session` block is
wrapped in a try/except that sets the Redis status to `error`. The CSV parsing and the
segment/vector creation that follow are not guarded, so an exception there skips the
`completed`/`error` update entirely. The initial `waiting` value is written with `setnx`
(no TTL) in `api/controllers/console/datasets/datasets_segments.py`, so it never expires.
### Environment
- Dify version: main (self-hosted / from source)
I'd like to work on this and will open a PR.
Contributor guide
Research direction
Start in api/tasks/batch_create_segment_to_index_task.py and trace the CSV parsing and segment/vector creation after the setup session block. Reproduce with an empty CSV or one-column QA CSV, then verify that failures set the Redis status to error and the status endpoint no longer leaves the UI polling indefinitely; inspect api/controllers/console/datasets/datasets_segments.py to understand the initial waiting value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- api, backend, databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100