docling-project / docling-project/docling

Footer Text Interferes with Main PDF Content During Parsing

Open
#1,625 3 comments 0 reactions 1 assignee Claimed by @vagenas View on GitHub
question
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
3d 4h
Merged PRs (30d)
95

Description

### Question

When parsing PDF documents using the doc_ling library, footer text (such as company names, copyright information, or page numbers) is being merged with the main body content. This causes issues particularly when the last line on a page is part of a sentence that continues on the next page. The footer text gets appended to this incomplete line, disrupting the logical flow and misleading downstream chunking or content interpretation.

This issue affects the semantic accuracy of the parsed content and can introduce noise in further processing tasks such as chunking or summarization.

Expected Behavior:
The parser should be able to detect and exclude footers from the main text content to preserve the integrity and continuity of the actual document body.

CODE:
from docling.chunking import HybridChunker
from langchain_docling import DoclingLoader

EMBED_MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2"
EXPORT_TYPE = ExportType.DOC_CHUNKS

loader = DoclingLoader(
file_path=FILE_PATH,
export_type=EXPORT_TYPE,
chunker=HybridChunker(tokenizer=EMBED_MODEL_ID),
)

d_docs = loader.load()

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.