docling-project / docling-project/docling-core

Global and per-page export_to_markdown are not byte-comparable when a list straddles a page boundary

Open
#710 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
282
Forks
214
Avg merge
2d 15h
Merged PRs (30d)
21

Description

### Bug

`export_to_markdown(page_break_placeholder=...)` and the per-page `export_to_markdown(page_no=n)` are the
two natural ways to get page-delimited markdown out of a `DoclingDocument`, and they agree on every input
I have tried **except one**: when a page boundary falls inside a `ListGroup`. There they differ by
whitespace, and **no join convention reconciles them**, because the difference is produced inside the
list rather than at the join:

- in the global export the break is emitted in the list's scope, and `MarkdownListSerializer` joins its
parts with `sep = "\n"` (`markdown.py:768`);
- the doc-level serializer joins with `"\n\n"` (`markdown.py:945`).

So joining the per-page exports with `"\n\n"` around the placeholder gives one string, joining with the
bare placeholder gives another, and the global export matches neither.

Nothing moves and nothing is lost: same text, same section per page, same number of placeholders. It is
cosmetic in the rendered output. What it breaks is using the per-page export as a **byte-exact** reference
for the global one — which is the first thing I would reach for in a test, and which works on every other
shape of document.

### Steps to reproduce

```python
from docling_core.types.doc.base import BoundingBox, CoordOrigin, Size
from docling_core.types.doc.document import DoclingDocument, PageItem, ProvenanceItem
from docling_core.types.doc.labels import DocItemLabel

PB = ""
BBOX = BoundingBox(l=50, t=700, r=500, b=680, coord_origin=CoordOrigin.BOTTOMLEFT)

doc = DoclingDocument(name="list_across_boundary")
for page_no in (1, 2):
doc.pages[page_no] = PageItem(page_no=page_no, size=Size(width=595, height=842))
doc.add_text(label=DocItemLabel.TEXT, text="Intro on page one",
prov=ProvenanceItem(page_no=1, bbox=BBOX, charspan=(0, 17)))
lst = doc.add_list_group(name="list")
for text, page_no in (("item one", 1), ("item two", 1), ("item three", 2), ("item four", 2)):
doc.add_list_item(text=text, parent=lst,
prov=ProvenanceItem(page_no=page_no, bbox=BBOX, charspan=(0, len(text))))

globally = doc.export_to_markdown(page_break_placeholder=PB)
per_page = f"\n\n{PB}\n\n".join(doc.export_to_markdown(page_no=n) for n in (1, 2))
print(repr(globally))
print(repr(per_page))
print(globally == per_page, globally.split() == per_page.split())
```

**Actual**

```
'Intro on page one\n\n- item one\n- item two\n\n- item three\n- item four'
'Intro on page one\n\n- item one\n- item two\n\n\n\n- item three\n- item four'
False True
```

Replace the list with plain text items and both forms come out byte-identical, which is what makes this
specific to the list scope rather than to page breaks in general.

**The two are structurally different, not just differently spaced**, and I think that is the real content
of the report. The per-page export yields two independent lists:

```
page 1 -> 'Intro on page one\n\n- item one\n- item two'
page 2 -> '- item three\n- item four'
```

while the global export yields **one** list with a break inside it. The whitespace is the symptom of that
choice, not the choice itself.

**Expected**: I don't think there is one obvious answer, so I'm reporting the discrepancy rather than
prescribing a fix. The options I can see:

1. the global export splits the list at the boundary, so both paths produce the same two lists — matches
the per-page export byte for byte, at the cost of representing one list as two;
2. the global export stays as it is, and the docs state that the per-page export is not byte-comparable
with it, so the right comparison is per-section content;
3. nothing changes and this sits here as a known asymmetry.

(2) is the cheapest and would have saved me the detour; (1) is the only one that makes byte equality
reachable. Either way, what I'd like to avoid is the next person concluding, as I briefly did, that a
2-byte difference means their page-break logic is wrong.

### Why I think it deserves its own issue

A test asserting byte equality between the two exports passes on every input until a list happens to
straddle a boundary, and then fails for a reason unrelated to what it was testing. That is how I ran into
it: verifying #708 against a 25-document corpus, one document failed a byte-exact comparison while all of
its 154 sections matched the reference word for word — the whole difference was two newlines.

### Not #705 / #708, and not #472 / #466

- #705 (fixed by #708) is the break emitted *after* a group, which misattributes content to the wrong
page. This one moves nothing.
- #472 / #466 is pages with no items producing no break. Unrelated.
- Reproduced identically on `v2.87.1`, a clean `v2.90.0` and #708 at `63af3fa`, so it predates and
survives that patch — @maxmilian also reproduced it independently in two worktrees
([#705 comment](https://github.com/docling-project/docling-core/issues/705#issuecomment-5193263739))
and confirmed `markdown.py` is untouched by #708.

### Docling version

`docling-core` 2.87.1 and 2.90.0 — same result on both, and on Python 3.10.12 and 3.12.10. Linux x86_64.

Contributor guide

Open the contributing guide

Research direction

Start with export_to_markdown and MarkdownListSerializer at the referenced locations in markdown.py, then run the provided two-page ListGroup reproduction. Compare the global and per-page outputs and determine which behavior should be supported. Done means the chosen behavior is covered by a regression test or documented as an intentional asymmetry.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
content
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.