docling-project / docling-project/docling-core

`HybridChunker._split_by_doc_items` re-serializes an `InlineGroup`'s runs independently, breaking inline formatting

Open
#770 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
282
Forks
214
Avg merge
2d 15h
Merged PRs (30d)
21

Description

`HierarchicalChunker` serializes an `InlineGroup` as one unit. A paragraph with inline markup chunks as `` Install with `pip install x` first. ``. `HybridChunker.chunk` then passes every chunk through `_split_by_doc_items`, which rebuilds each window with `_make_chunk_from_doc_items`. That method keeps the original text only when the chunk has a single `doc_item`. Otherwise it builds the text as

```python
self.delim.join(
res_text
for doc_item in doc_items
if (res_text := doc_serializer.serialize(item=doc_item).text)
and not isinstance(doc_item, TitleItem | SectionHeaderItem)
)
```

The `doc_items` of an inline-group chunk are its runs. Each run is serialized as a standalone item. A `TextItem` becomes its own line and a `CodeItem` becomes a fenced code block. The rebuild happens whether or not the chunk needs splitting, because `_split_by_doc_items` always reconstructs the window. It happens before peer merging, and the corruption occurs with `merge_peers=False`. The `TODO: merging should ideally be done by the serializer` above the join marks the spot.

Reproduced on docling-core 2.96.0 with the default tokenizer. The code is unchanged on `main`. This is distinct from #371 and PR #693, which concern the whitespace between runs when a group is serialized as one unit. Here the runs are not serialized as one unit at all.

## Reproduction

```python
from docling_core.types.doc.document import DoclingDocument
from docling_core.types.doc.labels import DocItemLabel
from docling_core.transforms.chunker.hierarchical_chunker import HierarchicalChunker
from docling_core.transforms.chunker.hybrid_chunker import HybridChunker

doc = DoclingDocument(name="t")
doc.add_title(text="Sample")
heading = doc.add_heading(text="", level=1)
group = doc.add_inline_group(parent=heading)
doc.add_text(label=DocItemLabel.TEXT, text="Release", parent=group, hyperlink="https://example.com/releases")
doc.add_text(label=DocItemLabel.TEXT, text="notes", parent=group)
para = doc.add_inline_group()
doc.add_text(label=DocItemLabel.TEXT, text="Install with", parent=para)
doc.add_code(text="pip install x", parent=para)
doc.add_text(label=DocItemLabel.TEXT, text="first.", parent=para)

for c in HierarchicalChunker().chunk(doc):
print("hierarchical: ", repr(c.text))
for c in HybridChunker(merge_peers=False).chunk(doc):
print("hybrid, no merging:", repr(c.text))
for c in HybridChunker().chunk(doc):
print("hybrid, default: ", repr(c.text))
```

Output on 2.96.0:

```
hierarchical: '[Release](https://example.com/releases) notes'
hierarchical: 'Install with `pip install x` first.'
hybrid, no merging: '[Release](https://example.com/releases)\nnotes'
hybrid, no merging: 'Install with\n```\npip install x\n```\nfirst.'
hybrid, default: '[Release](https://example.com/releases)\nnotes\nInstall with\n```\npip install x\n```\nfirst.'
```

A real document shows the same. docling 2.124.0's Markdown backend produces this shape for a paragraph mixing ordinary text with a code span. A paragraph that is only a code span is a bare `CodeItem`. A changelog converted and chunked with `HybridChunker` carries `By default,\n```\nhaiku.rag\n```\nuses the configured embedder.` where the source has `` By default, `haiku.rag` uses the configured embedder. ``

## Expected

A rebuilt window should render its items the way the serializer renders them together. The hierarchical chunks already contain that text. With this fixed alone:

```
hybrid, no merging: '[Release](https://example.com/releases) notes'
hybrid, no merging: 'Install with `pip install x` first.'
hybrid, default: '[Release](https://example.com/releases) notes\nInstall with `pip install x` first.'
```

The heading's runs appearing in the body at all is the separate `meta.headings` issue, #769. With both fixed, the first line disappears from the body and `Release notes` appears in the metadata.

Contributor guide

Open the contributing guide

Research direction

Start in docling_core/transforms/chunker/hybrid_chunker.py at HybridChunker.chunk, _split_by_doc_items, and the TODO above the join in _make_chunk_from_doc_items. Run the supplied reproduction with merge_peers=False and compare it with HierarchicalChunker output. Done means rebuilt windows preserve the serializer's combined inline formatting in both no-merging and default hybrid output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.