docling-project / docling-project/docling-core

`HybridChunker` downcasts `chunk.meta.doc_items` to bare `DocItem`

Open Beginner friendly
#728 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
282
Forks
214
Avg merge
2d 15h
Merged PRs (30d)
21

Description

`HybridChunker` returns chunks whose `meta.doc_items` have been downcast to bare `DocItem` instances. The (potential) concrete subclass is lost, and with it the item payload such as the `meta.description` of a `TableItem`. Note that `HierarchicalChunker` returns the document's own items.

The problematic lines are:
https://github.com/docling-project/docling-core/blob/dedc35da4c99e3ae597423358e5648eb29ee3cad/docling_core/transforms/chunker/hybrid_chunker.py#L240
https://github.com/docling-project/docling-core/blob/dedc35da4c99e3ae597423358e5648eb29ee3cad/docling_core/transforms/chunker/hybrid_chunker.py#L252

The former line has been present in the code from the very beginning (#68). I can only imagine that the idea was to create a (mutable) copy. A possible fix would be:
```diff
--- a/docling_core/transforms/chunker/hybrid_chunker.py
+++ b/docling_core/transforms/chunker/hybrid_chunker.py
@@ -237,7 +240,7 @@ class HybridChunker(BaseChunker):
) -> list[DocChunk]:
lengths = self._doc_chunk_length(doc_chunk)
if lengths.total_len <= self.max_tokens:
- return [DocChunk(**doc_chunk.export_json_dict())]
+ return [doc_chunk]
else:
# How much room is there for text after subtracting out the headers and
# captions:
@@ -249,7 +252,8 @@ class HybridChunker(BaseChunker):
"available size for the chunk, so they will be ignored: "
f"{doc_chunk.text=}, {doc_chunk.meta=}"
)
- new_chunk = DocChunk(**doc_chunk.export_json_dict())
+ new_chunk = doc_chunk.model_copy()
+ new_chunk.meta = doc_chunk.meta.model_copy()
new_chunk.meta.captions = None
new_chunk.meta.headings = None
```

---

Reproducible example is here:
```python
import tiktoken

from docling_core.transforms.chunker.hierarchical_chunker import HierarchicalChunker
from docling_core.transforms.chunker.hybrid_chunker import HybridChunker
from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer
from docling_core.types.doc import DocItemLabel, DoclingDocument
from docling_core.types.doc.common.meta import DescriptionMetaField, FloatingMeta
from docling_core.types.doc.document import TableCell, TableData

doc = DoclingDocument(name="repro")
doc.add_text(label=DocItemLabel.TEXT, text="Some paragraph.")
table = doc.add_table(
data=TableData(
num_rows=2,
num_cols=1,
table_cells=[
TableCell(text="Year", start_row_offset_idx=0, end_row_offset_idx=1,
start_col_offset_idx=0, end_col_offset_idx=1, column_header=True),
TableCell(text="2024", start_row_offset_idx=1, end_row_offset_idx=2,
start_col_offset_idx=0, end_col_offset_idx=1),
],
)
)
table.meta = FloatingMeta(description=DescriptionMetaField(text="Revenue by year."))

tokenizer = OpenAITokenizer(tokenizer=tiktoken.get_encoding("cl100k_base"), max_tokens=200)

for name, chunker in [
("HierarchicalChunker", HierarchicalChunker()),
("HybridChunker", HybridChunker(tokenizer=tokenizer)),
]:
items = [item for chunk in chunker.chunk(dl_doc=doc) for item in chunk.meta.doc_items]
print(f"{name:22} {[type(i).__name__ for i in items]}")
for item in items:
print(f"{'':22} text={getattr(item, 'text', '')!r} meta={getattr(item, 'meta', '')}")
```

The output I am getting is:
```
HierarchicalChunker ['TextItem', 'TableItem']
text='Some paragraph.' meta=None
text='' meta=summary=None language=None entities=None keywords=None topics=None description=DescriptionMetaField(confidence=None, created_by=None, text='Revenue by year.')
HybridChunker ['DocItem', 'DocItem']
text='' meta=None
text='' meta=summary=None language=None entities=None keywords=None topics=None
```

It shows that `HybridChunker` loses the information about the specific items, while `HierarchicalChunker` keeps it.

Contributor guide

Open the contributing guide

Research direction

Start in docling_core/transforms/chunker/hybrid_chunker.py at the lines linked around 240 and 252, then run the reproducible example from the issue. Verify that HybridChunker preserves concrete items such as TableItem and their metadata, matching the behavior shown for HierarchicalChunker.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.