docling-project / docling-project/docling-core
Global and per-page export_to_markdown are not byte-comparable when a list straddles a page boundary
- Dominant language
- HTML
- Stars
- 282
- Forks
- 214
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 21
Description
### Bug
`export_to_markdown(page_break_placeholder=...)` and the per-page `export_to_markdown(page_no=n)` are the
two natural ways to get page-delimited markdown out of a `DoclingDocument`, and they agree on every input
I have tried **except one**: when a page boundary falls inside a `ListGroup`. There they differ by
whitespace, and **no join convention reconciles them**, because the difference is produced inside the
list rather than at the join:
- in the global export the break is emitted in the list's scope, and `MarkdownListSerializer` joins its
parts with `sep = "\n"` (`markdown.py:768`);
- the doc-level serializer joins with `"\n\n"` (`markdown.py:945`).
So joining the per-page exports with `"\n\n"` around the placeholder gives one string, joining with the
bare placeholder gives another, and the global export matches neither.
Nothing moves and nothing is lost: same text, same section per page, same number of placeholders. It is
cosmetic in the rendered output. What it breaks is using the per-page export as a **byte-exact** reference
for the global one — which is the first thing I would reach for in a test, and which works on every other
shape of document.
### Steps to reproduce
```python
from docling_core.types.doc.base import BoundingBox, CoordOrigin, Size
from docling_core.types.doc.document import DoclingDocument, PageItem, ProvenanceItem
from docling_core.types.doc.labels import DocItemLabel
PB = ""
BBOX = BoundingBox(l=50, t=700, r=500, b=680, coord_origin=CoordOrigin.BOTTOMLEFT)
doc = DoclingDocument(name="list_across_boundary")
for page_no in (1, 2):
doc.pages[page_no] = PageItem(page_no=page_no, size=Size(width=595, height=842))
doc.add_text(label=DocItemLabel.TEXT, text="Intro on page one",
prov=ProvenanceItem(page_no=1, bbox=BBOX, charspan=(0, 17)))
lst = doc.add_list_group(name="list")
for text, page_no in (("item one", 1), ("item two", 1), ("item three", 2), ("item four", 2)):
doc.add_list_item(text=text, parent=lst,
prov=ProvenanceItem(page_no=page_no, bbox=BBOX, charspan=(0, len(text))))
globally = doc.export_to_markdown(page_break_placeholder=PB)
per_page = f"\n\n{PB}\n\n".join(doc.export_to_markdown(page_no=n) for n in (1, 2))
print(repr(globally))
print(repr(per_page))
print(globally == per_page, globally.split() == per_page.split())
```
**Actual**
```
'Intro on page one\n\n- item one\n- item two\n\n- item three\n- item four'
'Intro on page one\n\n- item one\n- item two\n\n\n\n- item three\n- item four'
False True
```
Replace the list with plain text items and both forms come out byte-identical, which is what makes this
specific to the list scope rather than to page breaks in general.
**The two are structurally different, not just differently spaced**, and I think that is the real content
of the report. The per-page export yields two independent lists:
```
page 1 -> 'Intro on page one\n\n- item one\n- item two'
page 2 -> '- item three\n- item four'
```
while the global export yields **one** list with a break inside it. The whitespace is the symptom of that
choice, not the choice itself.
**Expected**: I don't think there is one obvious answer, so I'm reporting the discrepancy rather than
prescribing a fix. The options I can see:
1. the global export splits the list at the boundary, so both paths produce the same two lists — matches
the per-page export byte for byte, at the cost of representing one list as two;
2. the global export stays as it is, and the docs state that the per-page export is not byte-comparable
with it, so the right comparison is per-section content;
3. nothing changes and this sits here as a known asymmetry.
(2) is the cheapest and would have saved me the detour; (1) is the only one that makes byte equality
reachable. Either way, what I'd like to avoid is the next person concluding, as I briefly did, that a
2-byte difference means their page-break logic is wrong.
### Why I think it deserves its own issue
A test asserting byte equality between the two exports passes on every input until a list happens to
straddle a boundary, and then fails for a reason unrelated to what it was testing. That is how I ran into
it: verifying #708 against a 25-document corpus, one document failed a byte-exact comparison while all of
its 154 sections matched the reference word for word — the whole difference was two newlines.
### Not #705 / #708, and not #472 / #466
- #705 (fixed by #708) is the break emitted *after* a group, which misattributes content to the wrong
page. This one moves nothing.
- #472 / #466 is pages with no items producing no break. Unrelated.
- Reproduced identically on `v2.87.1`, a clean `v2.90.0` and #708 at `63af3fa`, so it predates and
survives that patch — @maxmilian also reproduced it independently in two worktrees
([#705 comment](https://github.com/docling-project/docling-core/issues/705#issuecomment-5193263739))
and confirmed `markdown.py` is untouched by #708.
### Docling version
`docling-core` 2.87.1 and 2.90.0 — same result on both, and on Python 3.10.12 and 3.12.10. Linux x86_64.
Contributor guide
Research direction
Start with export_to_markdown and MarkdownListSerializer at the referenced locations in markdown.py, then run the provided two-page ListGroup reproduction. Compare the global and per-page outputs and determine which behavior should be supported. Done means the chosen behavior is covered by a regression test or documented as an intentional asymmetry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- content
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 48/100