docling-project / docling-project/docling
ReadingOrderModel produces invalid ProvenanceItem charspans when dehyphenating merged text elements
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Bug
`ReadingOrderModel._merge_elements()` computes the `ProvenanceItem.charspan` before applying dehyphenation. When the left text ends with either a soft hyphen (`\u00ad`) or a hard hyphen followed by a lowercase continuation, the merged text becomes 2 characters shorter than the span calculation assumes.
As a result, the provenance span can extend past `len(new_item.text)` and is therefore inconsistent with the resulting `TextItem.text`.
This is reproducible without any PDF input.
### Environment
- Docling: `2.123.1`
- docling-core: `2.93.0`
- Python: `3.13`
### Minimal reproducer
```python
from types import SimpleNamespace
from docling.models.stages.reading_order.readingorder_model import ReadingOrderModel
from docling_core.types.doc import BoundingBox, CoordOrigin
class DummyCluster:
bbox = BoundingBox(
l=10.0,
t=100.0,
r=200.0,
b=80.0,
coord_origin=CoordOrigin.BOTTOMLEFT,
)
class DummyElement:
def __init__(self, text, page_no=2, label="text"):
self.text = text
self.page_no = page_no
self.label = label
self.cluster = DummyCluster()
self.hyperlink = None
def run_case(name, left_text, right_text):
element = DummyElement(left_text, page_no=1)
merged_elem = DummyElement(right_text, page_no=2)
new_item = SimpleNamespace(
text=left_text,
orig=left_text,
label="text",
prov=[],
hyperlink=None,
)
ReadingOrderModel._merge_elements(
None,
element,
merged_elem,
new_item,
page_height=1000,
)
prov = new_item.prov[-1]
start, end = prov.charspan
print(name)
print("result:", repr(new_item.text))
print("result length:", len(new_item.text))
print("provenance span:", prov.charspan)
print("span valid:", 0 <= start < end <= len(new_item.text))
print("end - text_len:", end - len(new_item.text))
print()
run_case(
"NORMAL MERGE",
"Das ist die erste Zeile",
"und hier geht es weiter.",
)
run_case(
"SOFT-HYPHEN MERGE",
"Das ist ein Rechenzent\u00ad",
"rum mit hoher Verfügbarkeit.",
)
run_case(
"HARD-HYPHEN LOWERCASE MERGE",
"Das ist ein Rechenzent-",
"rum mit hoher Verfügbarkeit.",
)
```
### Observed output
```text
NORMAL MERGE
result length: 48
provenance span: (24, 48)
span valid: True
end - text_len: 0
SOFT-HYPHEN MERGE
result length: 50
provenance span: (24, 52)
span valid: False
end - text_len: 2
HARD-HYPHEN LOWERCASE MERGE
result length: 50
provenance span: (24, 52)
span valid: False
end - text_len: 2
```
### Suspected cause
In the current implementation, the provenance span is calculated first using:
```python
charspan=(
len(new_item.text) + 1,
len(new_item.text) + 1 + len(merged_elem.text),
)
```
After that, dehyphenation may execute:
```python
new_item.text = new_item.text[:-1] + merged_elem.text
```
For the dehyphenation path, the actual merged text is therefore 2 characters shorter than the span calculation assumes: the trailing hyphen is removed and the separating space is not inserted.
### Real-world impact
This also occurs on real multi-page PDF text items. In one document, a `TextItem.text` of length `725` had a final provenance span ending at `727`, with overlapping adjacent spans. In another large document, the same `+2` pattern appeared repeatedly across many multi-page text items.
Downstream systems that validate provenance against the final `TextItem.text` can therefore correctly reject these chunks as inconsistent.
### Related issue
This seems related to the same merge/provenance area as #1699, but it is a different failure mode. #1699 concerned incorrect page/bbox population and was resolved in newer releases; the charspan/dehyphenation inconsistency above is still reproducible in `2.123.1`.
### Expected behavior
After any merge path, every `ProvenanceItem.charspan` should remain valid for the resulting `TextItem.text`, i.e. `0 <= start < end <= len(text)`, and adjacent spans should reflect the actual merged text layout without introducing artificial overlap or out-of-range offsets.
Contributor guide
Research direction
Start in docling/models/stages/reading_order/readingorder_model.py at ReadingOrderModel._merge_elements(), then run the supplied reproducer for normal, soft-hyphen, and hard-hyphen merges. Done means every resulting ProvenanceItem.charspan is within the final TextItem.text length and adjacent spans match the merged text without artificial overlap.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100