langgenius / langgenius/dify

[Refactor/Chore] Batch-load dataset document list response details

Open
#40,123 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for refactors or chores; if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.

### Description

The console dataset document list endpoint resolves several response fields—hit counts, data-source details, process rules, and metadata—through per-document database lookups. With a full page, the number of queries grows linearly with the page size. When `fetch=true`, completed and total segment counts add two more queries per document.

### Motivation

Batch the related rows for all documents on the current page and reuse them during response serialization so the query count remains bounded. Preserve the existing response shape and counting semantics, including re-segment handling and tenant/dataset ownership boundaries.

### Additional Context

Target endpoint: `GET /console/api/datasets/{dataset_id}/documents`.

Add regression coverage that compares the SELECT count for one document and multiple documents, along with response-value checks.

Contributor guide

Open the contributing guide

Research direction

Start by tracing GET /console/api/datasets/{dataset_id}/documents and identify where response fields and fetch=true segment counts trigger per-document lookups. Add regression coverage comparing SELECT counts for one versus multiple documents, while checking response values, re-segment handling, and tenant/dataset ownership boundaries. Done means the response shape and counting semantics are preserved with bounded query growth.

Written by the indexing model from the issue text.

Assessment

Domain
backend-api-design, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.