docling-project / docling-project/docling

param "page_break_placeholder" didn't work on HybridChunker handling DoclingDocument

Open
#2,840 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug
I gonna to chunk document with HybridChunker , and needing page_break_placeholder flag so that i can do more detailed customized chunk after HybridChunker chunking.

### Steps to reproduce
def main():
converter = DocumentConverter()
result = converter.convert(source="tmp/Payroll.pptx")
# md = result.document.export_to_markdown(page_break_placeholder="")

tokenizer = OpenAITokenizer(
tokenizer=tiktoken.get_encoding("cl100k_base"),
max_tokens=1024
)

class MDTableSerializerProvider(ChunkingSerializerProvider):
def get_serializer(self, doc):
md_serializer=ChunkingDocSerializer(
doc=doc
)
md_serializer.params.page_break_placeholder=""
return md_serializer

chunker = HybridChunker(
tokenizer=tokenizer,
merge_peers=True,
serializer_provider=MDTableSerializerProvider(), # optional, defaults to True
)
chunk_iter = chunker.chunk(dl_doc=result.document)
chunks = list(chunk_iter)
for i, chunk in enumerate(chunks):
print(f"Chunk {i}: {chunk.text}")

if __name__ == "__main__":
main()

### Docling version
2.61.2

### Python version
3.13.8

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.