docling-project / docling-project/docling

ReadingOrderModel produces invalid ProvenanceItem charspans when dehyphenating merged text elements

Open Beginner friendly
#4,217 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug

`ReadingOrderModel._merge_elements()` computes the `ProvenanceItem.charspan` before applying dehyphenation. When the left text ends with either a soft hyphen (`\u00ad`) or a hard hyphen followed by a lowercase continuation, the merged text becomes 2 characters shorter than the span calculation assumes.

As a result, the provenance span can extend past `len(new_item.text)` and is therefore inconsistent with the resulting `TextItem.text`.

This is reproducible without any PDF input.

### Environment

- Docling: `2.123.1`
- docling-core: `2.93.0`
- Python: `3.13`

### Minimal reproducer

```python
from types import SimpleNamespace

from docling.models.stages.reading_order.readingorder_model import ReadingOrderModel
from docling_core.types.doc import BoundingBox, CoordOrigin

class DummyCluster:
bbox = BoundingBox(
l=10.0,
t=100.0,
r=200.0,
b=80.0,
coord_origin=CoordOrigin.BOTTOMLEFT,
)

class DummyElement:
def __init__(self, text, page_no=2, label="text"):
self.text = text
self.page_no = page_no
self.label = label
self.cluster = DummyCluster()
self.hyperlink = None

def run_case(name, left_text, right_text):
element = DummyElement(left_text, page_no=1)
merged_elem = DummyElement(right_text, page_no=2)

new_item = SimpleNamespace(
text=left_text,
orig=left_text,
label="text",
prov=[],
hyperlink=None,
)

ReadingOrderModel._merge_elements(
None,
element,
merged_elem,
new_item,
page_height=1000,
)

prov = new_item.prov[-1]
start, end = prov.charspan

print(name)
print("result:", repr(new_item.text))
print("result length:", len(new_item.text))
print("provenance span:", prov.charspan)
print("span valid:", 0 <= start < end <= len(new_item.text))
print("end - text_len:", end - len(new_item.text))
print()

run_case(
"NORMAL MERGE",
"Das ist die erste Zeile",
"und hier geht es weiter.",
)

run_case(
"SOFT-HYPHEN MERGE",
"Das ist ein Rechenzent\u00ad",
"rum mit hoher Verfügbarkeit.",
)

run_case(
"HARD-HYPHEN LOWERCASE MERGE",
"Das ist ein Rechenzent-",
"rum mit hoher Verfügbarkeit.",
)
```

### Observed output

```text
NORMAL MERGE
result length: 48
provenance span: (24, 48)
span valid: True
end - text_len: 0

SOFT-HYPHEN MERGE
result length: 50
provenance span: (24, 52)
span valid: False
end - text_len: 2

HARD-HYPHEN LOWERCASE MERGE
result length: 50
provenance span: (24, 52)
span valid: False
end - text_len: 2
```

### Suspected cause

In the current implementation, the provenance span is calculated first using:

```python
charspan=(
len(new_item.text) + 1,
len(new_item.text) + 1 + len(merged_elem.text),
)
```

After that, dehyphenation may execute:

```python
new_item.text = new_item.text[:-1] + merged_elem.text
```

For the dehyphenation path, the actual merged text is therefore 2 characters shorter than the span calculation assumes: the trailing hyphen is removed and the separating space is not inserted.

### Real-world impact

This also occurs on real multi-page PDF text items. In one document, a `TextItem.text` of length `725` had a final provenance span ending at `727`, with overlapping adjacent spans. In another large document, the same `+2` pattern appeared repeatedly across many multi-page text items.

Downstream systems that validate provenance against the final `TextItem.text` can therefore correctly reject these chunks as inconsistent.

### Related issue

This seems related to the same merge/provenance area as #1699, but it is a different failure mode. #1699 concerned incorrect page/bbox population and was resolved in newer releases; the charspan/dehyphenation inconsistency above is still reproducible in `2.123.1`.

### Expected behavior

After any merge path, every `ProvenanceItem.charspan` should remain valid for the resulting `TextItem.text`, i.e. `0 <= start < end <= len(text)`, and adjacent spans should reflect the actual merged text layout without introducing artificial overlap or out-of-range offsets.

Contributor guide

Open the contributing guide

Research direction

Start in docling/models/stages/reading_order/readingorder_model.py at ReadingOrderModel._merge_elements(), then run the supplied reproducer for normal, soft-hyphen, and hard-hyphen merges. Done means every resulting ProvenanceItem.charspan is within the final TextItem.text length and adjacent spans match the merged text without artificial overlap.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.