docling-project / docling-project/docling-core

Maximum length for the markdown exporter

Open
#33 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
HTML
Stars
282
Forks
214
Avg merge
2d 15h
Merged PRs (30d)
21

Description

When stuffing content to LLMs you want to get the biggest chunk that fits in the maximum context window.

A trivial implementation could simply do `content_md = doc.export_to_markdown()[:MAX_LEN]`. However, this could break sentences, paragraphs, tables, etc.

If we introduce a `max_size` in the `export_to_markdown()` function, we could control it by adding a document item only if it doesn't go above the limit.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.