docling-project / docling-project/docling
Footer Text Interferes with Main PDF Content During Parsing
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 95
Description
### Question
When parsing PDF documents using the doc_ling library, footer text (such as company names, copyright information, or page numbers) is being merged with the main body content. This causes issues particularly when the last line on a page is part of a sentence that continues on the next page. The footer text gets appended to this incomplete line, disrupting the logical flow and misleading downstream chunking or content interpretation.
This issue affects the semantic accuracy of the parsed content and can introduce noise in further processing tasks such as chunking or summarization.
Expected Behavior:
The parser should be able to detect and exclude footers from the main text content to preserve the integrity and continuity of the actual document body.
CODE:
from docling.chunking import HybridChunker
from langchain_docling import DoclingLoader
EMBED_MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2"
EXPORT_TYPE = ExportType.DOC_CHUNKS
loader = DoclingLoader(
file_path=FILE_PATH,
export_type=EXPORT_TYPE,
chunker=HybridChunker(tokenizer=EMBED_MODEL_ID),
)
d_docs = loader.load()
Contributor guide
Assessment
This issue has not been assessed yet.