docling-project / docling-project/docling-core
`HybridChunker._split_by_doc_items` re-serializes an `InlineGroup`'s runs independently, breaking inline formatting
- Dominant language
- HTML
- Stars
- 282
- Forks
- 214
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 21
Description
`HierarchicalChunker` serializes an `InlineGroup` as one unit. A paragraph with inline markup chunks as `` Install with `pip install x` first. ``. `HybridChunker.chunk` then passes every chunk through `_split_by_doc_items`, which rebuilds each window with `_make_chunk_from_doc_items`. That method keeps the original text only when the chunk has a single `doc_item`. Otherwise it builds the text as
```python
self.delim.join(
res_text
for doc_item in doc_items
if (res_text := doc_serializer.serialize(item=doc_item).text)
and not isinstance(doc_item, TitleItem | SectionHeaderItem)
)
```
The `doc_items` of an inline-group chunk are its runs. Each run is serialized as a standalone item. A `TextItem` becomes its own line and a `CodeItem` becomes a fenced code block. The rebuild happens whether or not the chunk needs splitting, because `_split_by_doc_items` always reconstructs the window. It happens before peer merging, and the corruption occurs with `merge_peers=False`. The `TODO: merging should ideally be done by the serializer` above the join marks the spot.
Reproduced on docling-core 2.96.0 with the default tokenizer. The code is unchanged on `main`. This is distinct from #371 and PR #693, which concern the whitespace between runs when a group is serialized as one unit. Here the runs are not serialized as one unit at all.
## Reproduction
```python
from docling_core.types.doc.document import DoclingDocument
from docling_core.types.doc.labels import DocItemLabel
from docling_core.transforms.chunker.hierarchical_chunker import HierarchicalChunker
from docling_core.transforms.chunker.hybrid_chunker import HybridChunker
doc = DoclingDocument(name="t")
doc.add_title(text="Sample")
heading = doc.add_heading(text="", level=1)
group = doc.add_inline_group(parent=heading)
doc.add_text(label=DocItemLabel.TEXT, text="Release", parent=group, hyperlink="https://example.com/releases")
doc.add_text(label=DocItemLabel.TEXT, text="notes", parent=group)
para = doc.add_inline_group()
doc.add_text(label=DocItemLabel.TEXT, text="Install with", parent=para)
doc.add_code(text="pip install x", parent=para)
doc.add_text(label=DocItemLabel.TEXT, text="first.", parent=para)
for c in HierarchicalChunker().chunk(doc):
print("hierarchical: ", repr(c.text))
for c in HybridChunker(merge_peers=False).chunk(doc):
print("hybrid, no merging:", repr(c.text))
for c in HybridChunker().chunk(doc):
print("hybrid, default: ", repr(c.text))
```
Output on 2.96.0:
```
hierarchical: '[Release](https://example.com/releases) notes'
hierarchical: 'Install with `pip install x` first.'
hybrid, no merging: '[Release](https://example.com/releases)\nnotes'
hybrid, no merging: 'Install with\n```\npip install x\n```\nfirst.'
hybrid, default: '[Release](https://example.com/releases)\nnotes\nInstall with\n```\npip install x\n```\nfirst.'
```
A real document shows the same. docling 2.124.0's Markdown backend produces this shape for a paragraph mixing ordinary text with a code span. A paragraph that is only a code span is a bare `CodeItem`. A changelog converted and chunked with `HybridChunker` carries `By default,\n```\nhaiku.rag\n```\nuses the configured embedder.` where the source has `` By default, `haiku.rag` uses the configured embedder. ``
## Expected
A rebuilt window should render its items the way the serializer renders them together. The hierarchical chunks already contain that text. With this fixed alone:
```
hybrid, no merging: '[Release](https://example.com/releases) notes'
hybrid, no merging: 'Install with `pip install x` first.'
hybrid, default: '[Release](https://example.com/releases) notes\nInstall with `pip install x` first.'
```
The heading's runs appearing in the body at all is the separate `meta.headings` issue, #769. With both fixed, the first line disappears from the body and `Release notes` appears in the metadata.
Contributor guide
Research direction
Start in docling_core/transforms/chunker/hybrid_chunker.py at HybridChunker.chunk, _split_by_doc_items, and the TODO above the join in _make_chunk_from_doc_items. Run the supplied reproduction with merge_peers=False and compare it with HierarchicalChunker output. Done means rebuilt windows preserve the serializer's combined inline formatting in both no-merging and default hybrid output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100