docling-project / docling-project/docling-core

Page-break marker leaks the internal sentinel when a single node is serialized (and into chunk text)

Open
#714 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
282
Forks
214
Avg merge
2d 15h
Merged PRs (30d)
21

Description

### Bug

When `page_break_placeholder` (Markdown) or `add_page_break` (DocTags) is enabled, a page break that falls **inside a group** leaks the serializer's internal sentinel — `#_#_DOCLING_DOC_PAGE_BREAK___#_#` — into the output of any call that is not the whole-document one.

The sentinel is produced in `DocSerializer.serialize()` ([`common.py:481`](https://github.com/docling-project/docling-core/blob/v2.91.0/docling_core/transforms/serializer/common.py#L481)) and is only ever resolved in `serialize_doc()`, which the single-node path never reaches. So `serializer.serialize(item=...)` returns it verbatim, and so does anything built on top of it — including `HierarchicalChunker`, which puts the raw sentinel into `chunk.text`.

### Steps to reproduce

```python
from docling_core.types.doc.base import BoundingBox, CoordOrigin, Size
from docling_core.types.doc.document import DoclingDocument, PageItem, ProvenanceItem
from docling_core.transforms.serializer.base import BaseSerializerProvider
from docling_core.transforms.serializer.markdown import MarkdownDocSerializer, MarkdownParams
from docling_core.transforms.chunker.hierarchical_chunker import (
ChunkingDocSerializer,
HierarchicalChunker,
)

BBOX = BoundingBox(l=50, t=700, r=500, b=680, coord_origin=CoordOrigin.BOTTOMLEFT)
prov = lambda pn, t: ProvenanceItem(page_no=pn, bbox=BBOX, charspan=(0, len(t)))

doc = DoclingDocument(name="d")
for pn in (1, 2):
doc.pages[pn] = PageItem(page_no=pn, size=Size(width=595, height=842))
lst = doc.add_list_group(name="l")
for text, pn in (("item one", 1), ("item two", 2)):
doc.add_list_item(text=text, parent=lst, prov=prov(pn, text))

ser = MarkdownDocSerializer(
doc=doc, params=MarkdownParams(page_break_placeholder="")
)
print("whole document:", repr(ser.serialize().text))
print("single group :", repr(ser.serialize(item=lst).text))

class P(BaseSerializerProvider):
def get_serializer(self, doc):
return ChunkingDocSerializer(
doc=doc, params=MarkdownParams(page_break_placeholder="")
)

for chunk in HierarchicalChunker(serializer_provider=P()).chunk(dl_doc=doc):
print("chunk :", repr(chunk.text))
```

**Actual**

```text
whole document: '- item one\n\n- item two'
single group : '- item one\n#_#_DOCLING_DOC_PAGE_BREAK_1_2_#_#\n- item two'
chunk : '- item one\n#_#_DOCLING_DOC_PAGE_BREAK_1_2_#_#\n- item two'
```

**Expected** — the same page-break rendering on all three paths:

```text
single group : '- item one\n\n- item two'
chunk : '- item one\n\n- item two'
```

DocTags behaves identically: `DocTagsDocSerializer(doc=doc).serialize(item=lst).text` yields
`…#_#_DOCLING_DOC_PAGE_BREAK_1_2_#_#…` instead of ``.

### Scope

The marker only escapes when it is nested inside a group. A page break between two top-level items lands directly in the `parts` list handed to `serialize_doc()`, which resolves it — so the whole-document output is correct in every case, and only the single-node entry point is affected. That is also why this survives a full-document golden-file suite: I compared the Markdown (`page_break_placeholder` set and empty), DocTags, LaTeX and HTML `SPLIT_PAGE` exports of all 38 documents under `test/data/doc/` — 190 whole-document outputs, every one byte-identical with and without the fix.

Of the four serializers that resolve page breaks, this affects the three whose replacement is context-free:

| serializer | replacement | affected |
|---|---|---|
| Markdown | `page_break_placeholder` | yes |
| DocTags | `` | yes |
| LaTeX | `page_break_command` | yes |
| HTML | `HTMLOutputStyle.SPLIT_PAGE` | needs `prev_page`/`next_page` and the marker's offset, so it can only be resolved once the whole document is available |

### Docling version

`docling-core` 2.91.0, reproduced on `main` at `2cae21e`. Python 3.12, Windows and Linux.

Contributor guide

Open the contributing guide

Research direction

Start at DocSerializer.serialize() in common.py:481 and compare the single-node path with serialize_doc(), then run the Markdown reproduction involving HierarchicalChunker. Verify that single-group serialization and chunk.text resolve page breaks like whole-document output for Markdown, DocTags, and LaTeX, without changing the existing HTML limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.