docling-project / docling-project/docling-core
Page-break marker leaks the internal sentinel when a single node is serialized (and into chunk text)
- Dominant language
- HTML
- Stars
- 282
- Forks
- 214
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 21
Description
### Bug
When `page_break_placeholder` (Markdown) or `add_page_break` (DocTags) is enabled, a page break that falls **inside a group** leaks the serializer's internal sentinel — `#_#_DOCLING_DOC_PAGE_BREAK___#_#` — into the output of any call that is not the whole-document one.
The sentinel is produced in `DocSerializer.serialize()` ([`common.py:481`](https://github.com/docling-project/docling-core/blob/v2.91.0/docling_core/transforms/serializer/common.py#L481)) and is only ever resolved in `serialize_doc()`, which the single-node path never reaches. So `serializer.serialize(item=...)` returns it verbatim, and so does anything built on top of it — including `HierarchicalChunker`, which puts the raw sentinel into `chunk.text`.
### Steps to reproduce
```python
from docling_core.types.doc.base import BoundingBox, CoordOrigin, Size
from docling_core.types.doc.document import DoclingDocument, PageItem, ProvenanceItem
from docling_core.transforms.serializer.base import BaseSerializerProvider
from docling_core.transforms.serializer.markdown import MarkdownDocSerializer, MarkdownParams
from docling_core.transforms.chunker.hierarchical_chunker import (
ChunkingDocSerializer,
HierarchicalChunker,
)
BBOX = BoundingBox(l=50, t=700, r=500, b=680, coord_origin=CoordOrigin.BOTTOMLEFT)
prov = lambda pn, t: ProvenanceItem(page_no=pn, bbox=BBOX, charspan=(0, len(t)))
doc = DoclingDocument(name="d")
for pn in (1, 2):
doc.pages[pn] = PageItem(page_no=pn, size=Size(width=595, height=842))
lst = doc.add_list_group(name="l")
for text, pn in (("item one", 1), ("item two", 2)):
doc.add_list_item(text=text, parent=lst, prov=prov(pn, text))
ser = MarkdownDocSerializer(
doc=doc, params=MarkdownParams(page_break_placeholder="")
)
print("whole document:", repr(ser.serialize().text))
print("single group :", repr(ser.serialize(item=lst).text))
class P(BaseSerializerProvider):
def get_serializer(self, doc):
return ChunkingDocSerializer(
doc=doc, params=MarkdownParams(page_break_placeholder="")
)
for chunk in HierarchicalChunker(serializer_provider=P()).chunk(dl_doc=doc):
print("chunk :", repr(chunk.text))
```
**Actual**
```text
whole document: '- item one\n\n- item two'
single group : '- item one\n#_#_DOCLING_DOC_PAGE_BREAK_1_2_#_#\n- item two'
chunk : '- item one\n#_#_DOCLING_DOC_PAGE_BREAK_1_2_#_#\n- item two'
```
**Expected** — the same page-break rendering on all three paths:
```text
single group : '- item one\n\n- item two'
chunk : '- item one\n\n- item two'
```
DocTags behaves identically: `DocTagsDocSerializer(doc=doc).serialize(item=lst).text` yields
`…#_#_DOCLING_DOC_PAGE_BREAK_1_2_#_#…` instead of ``.
### Scope
The marker only escapes when it is nested inside a group. A page break between two top-level items lands directly in the `parts` list handed to `serialize_doc()`, which resolves it — so the whole-document output is correct in every case, and only the single-node entry point is affected. That is also why this survives a full-document golden-file suite: I compared the Markdown (`page_break_placeholder` set and empty), DocTags, LaTeX and HTML `SPLIT_PAGE` exports of all 38 documents under `test/data/doc/` — 190 whole-document outputs, every one byte-identical with and without the fix.
Of the four serializers that resolve page breaks, this affects the three whose replacement is context-free:
| serializer | replacement | affected |
|---|---|---|
| Markdown | `page_break_placeholder` | yes |
| DocTags | `` | yes |
| LaTeX | `page_break_command` | yes |
| HTML | `HTMLOutputStyle.SPLIT_PAGE` | needs `prev_page`/`next_page` and the marker's offset, so it can only be resolved once the whole document is available |
### Docling version
`docling-core` 2.91.0, reproduced on `main` at `2cae21e`. Python 3.12, Windows and Linux.
Contributor guide
Research direction
Start at DocSerializer.serialize() in common.py:481 and compare the single-node path with serialize_doc(), then run the Markdown reproduction involving HierarchicalChunker. Verify that single-group serialization and chunk.text resolve page breaks like whole-document output for Markdown, DocTags, and LaTeX, without changing the existing HTML limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100