infiniflow / infiniflow/ragflow
TokenChunker text-path children split sets 'mom' to the wrong parent segment
- Dominant language
- Go
- Stars
- 91k
- Forks
- 10.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 705
Description
## Problem
During review of #17868 a `mom` semantics divergence was found on the **text/markdown/html path** of `TokenChunker` (`internal/ingestion/component/chunker/token.go`).
`invokeTextPayload` forks on whether a primary delimiter is active (`token.go:306`):
- **Branch B** (active primary delimiter) → `applyChildrenDelim` (`token.go:1209`) sets `Mom: seg`, i.e. the primary-split segment. This matches Python (`token_chunker.py:330-339`, `"mom": c`).
- **Branch A** (no active primary delimiter — e.g. default `delimiter_mode="token_size"` + `children_delimiters` set) → `mergeByTokenSize` (`token.go:554`) builds token-merged chunks, then `applyChildrenDelimText` (`token.go:1236`, called at `token.go:672`) sets `Mom: t` where `t = d.Text`, i.e. the **token-merged chunk** produced by `mergeByTokenSize`.
Python always sets `mom` to the parent segment `c` it split children from (`_split_text_by_pattern(c, custom_pattern)` → `"mom": c`). In default mode `c` comes from `naive_merge`. Go's `mergeByTokenSize` uses `sentenceDelimiter` + greedy token merge, a **different algorithm from Python's `naive_merge`**, so the parent unit differs and `mom` content/boundaries diverge (only visible when the text spans multiple chunks; single-chunk text happens to match).
Secondary fragility: `applyChildrenDelimText` re-derives `mom` from `d.Text` instead of preserving any incoming `Mom` (unlike `applyChildrenDelim`, which takes the parent as an explicit `seg` arg). A doc that already carried a meaningful `Mom` flowing into `applyChildrenDelimText` would have it silently overwritten.
The JSON path (`splitByChildren`, `token.go:733`) is unaffected — its `mom = ck.Text` (the item text) matches Python.
## Impact
- Parity harness flags `mom` diffs for text-path children cases that span multiple chunks.
- Downstream logic relying on `mom` as the child's parent-context pointer gets a Go-specific (different-granularity) segment vs Python. Embeddings are unaffected (`mom` is typically not embedded), but reassembly / parent-context display can diverge.
- Latent `Mom` overwrite in `applyChildrenDelimText`.
## Correct behavior
Every child's `mom` must equal the exact parent segment it was split from, at the same granularity/boundaries as Python's `naive_merge` segment `c` (default mode) or primary-split segment (delimiter mode).
## Proposed fix
1. Low-risk (do first): make `applyChildrenDelimText` preserve any incoming `Mom`, falling back to `d.Text` only when empty — symmetric with `applyChildrenDelim`:
```go
mom := d.Mom
if mom == "" {
mom = d.Text
}
out = append(out, schema.ChunkDoc{Text: child, Mom: mom})
```
2. Real alignment (larger effort, separate PR): align the Branch A merge with Python's `naive_merge` so children split operates on parent segments of the same granularity as Python. This is part of the broader Go↔Python merge-alignment work, not in scope for #17868.
## Scope
Go-side only. Python (`rag/flow/chunker/token_chunker.py`) is the reference and is correct; it must stay untouched. This divergence is pre-existing and unrelated to #17868 (which only changed delimiter retention in `text`, not `mom`).
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.