docling-project / docling-project/docling

[Bee] Make function 'concatenate(docs)' keep the original page no to support page range

Open
#3,890 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
3d 4h
Merged PRs (30d)
95

Description

### Requested feature
For multi-page PDF processing, sometimes we wanna split the file into several chunks to do parallel processing, and then merge the docling document at the end. If the file includes multiple tables, and the function `add_document()` is used, sometimes I got the error
```
Document hierarchy is inconsistent. #/tables/14 has cell #/groups/0 with parent #/tables/3 [type=value_error, input_value=DoclingDocument(schema_na...age=None, page_no=170)}), input_type=DoclingDocument]
```
To solve this issue, the function `concatenate(docs)` is used. However, it has one limitation that cannot keep the original page number. If the processing is just part of the file based on the page range, for example, page 5 - 20, and the chunks are page (5-10), (11-20), after calling concatenate(docs) to merge the Docling docs, the page no will be reset from 1, not keeping the starting page from 5. I have to calculate the page offset and update the page no.

The code I added to update the page no to support page range:
```
def update_docling_doc_page_no(self, merged_doc: DoclingDocument, page_offset: int):
if page_offset < 1:
return
# 1. Update pages dict keys and PageItem.page_no
new_pages = {}
for old_no, page_item in merged_doc.pages.items():
new_no = old_no + page_offset
page_item.page_no = new_no
new_pages[new_no] = page_item
merged_doc.pages = new_pages

# 2. Update ProvenanceItem.page_no on all content items
for item, _level in merged_doc.iterate_items(with_groups=False):
if hasattr(item, "prov"):
for prov in item.prov:
prov.page_no += page_offset

# 3. Update GraphCell.prov.page_no in KeyValueItem / FormItem
if isinstance(item, (KeyValueItem, FormItem)):
for cell in item.graph.cells:
if cell.prov is not None:
cell.prov.page_no += page_offset
```

### Samples of Docling docs that got the "Document hierarchy is inconsistent." error
- Function used: add_document()
- file:
[0704.3575_Averages_of_b-hadron_properties_at_the_end_of_2006_11_pages.pdf](https://github.com/user-attachments/files/30402725/0704.3575_Averages_of_b-hadron_properties_at_the_end_of_2006_11_pages.pdf)
- Chunks: page (1-10), page(11)
- Docling docs:
[segment_0.json](https://github.com/user-attachments/files/30402741/segment_0.json)
[segment_1.json](https://github.com/user-attachments/files/30402740/segment_1.json)

Contributor guide

Open the contributing guide

Research direction

Start at the implementation of concatenate(docs) and inspect how merged pages and provenance are numbered; the issue also shows add_document() as the affected usage. Done means concatenating chunks from an original page range preserves the original page numbers throughout the document, including page and provenance data, without causing the reported hierarchy inconsistency.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.