langgenius / langgenius/dify

Keyword search silently returns no results for parent-child chunks

Open
#40,680 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

## Self Checks

- [x] I have read the Contributing Guide and Language Policy.
- [x] This report describes a reproducible correctness bug.
- [x] I have searched existing issues, including closed issues.
- [x] I am submitting this report in English.
- [x] I have kept the bug-report fields and supplied the required information.

## Dify version

`main` at `616736e0f1585774dc394655bca52d5841a198ca` (the bug is also reproducible on later `main` revisions before the proposed fix).

## Cloud or Self Hosted

- [x] Self Hosted (Docker)
- [x] Self Hosted (Source)

## Steps to reproduce

1. Create a hierarchical dataset containing a parent segment and child chunks.
2. Build a keyword index for the child documents using their `ChildChunk.index_node_id` values.
3. Run keyword retrieval with a term extracted from a child chunk.
4. Observe that the keyword table finds the child node ID, but the search result is empty.

### Root cause

`Jieba.search()` resolves keyword hits only through `DocumentSegment.index_node_id`. Parent-child keyword entries use `ChildChunk.index_node_id`, so a valid hit can never be materialized as a retrieval document.

The lifecycle is also incomplete: the initial economy indexing path skips parent-child documents, and child create/update/delete/cleanup paths do not consistently synchronize the keyword table.

This is a silent correctness failure: no exception or warning is emitted, so users see an ordinary empty result even when the matching child chunk and keyword entry exist.

## Expected Behavior

- Parent-child economy indexing should add child node IDs to the keyword table.
- Child create, update, delete, and cleanup operations should keep the keyword table synchronized.
- `Jieba.search()` should resolve matching IDs from both `ChildChunk` and `DocumentSegment`.
- Child results should preserve ranking order and `document_ids_filter`, and retain the metadata needed to return the parent segment as context.

## Actual Behavior

Child node IDs may exist in the keyword table, but `Jieba.search()` queries only `DocumentSegment`, so the matching records are silently dropped and retrieval returns no results.

## Additional context

`ChildChunk` has no denormalized `keywords` column. Child keywords are stored in the existing `DatasetKeywordTable`, so no database migration is required.

Existing hierarchical datasets that rely on the keyword store need a keyword-index rebuild before historical child chunks become searchable.

Related reports describe missing exact-term retrieval, but do not identify this keyword-store/child-record mismatch:

- https://github.com/langgenius/dify/issues/35526
- https://github.com/langgenius/dify/issues/30736

Contributor guide

Open the contributing guide

Research direction

Start at Jieba.search() and trace how DatasetKeywordTable IDs are resolved against DocumentSegment and ChildChunk. Then inspect the economy indexing and child create, update, delete, and cleanup paths described in the report. Done means parent-child keyword entries stay synchronized, child hits preserve ranking and document_ids_filter, and matching results retain parent context; rebuild existing indexes when validating historical data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.