docling-project / docling-project/docling
Dehyphenation merges line-final hyphens that belong to the word, losing them irrecoverably (regression in 2.118.0)
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Bug
When a hyphenated word is broken across a line, Docling joins the halves and deletes the hyphen without checking whether the hyphen is part of the word. n-tridecanal becomes ntridecanal.
Mid-line hyphens are handled correctly, so the logic is already position-aware. The gap is that a line-final hyphen can also be a real hyphen.
The loss is unrecoverable: once the hyphen is gone, nothing downstream can tell whether the page printed ntridecanal or n-tridecanal. The opposite failure — leaving n- tridecanal unjoined, which 2.107.0 does — is wrong but repairable by anyone later.
### Steps to reproduce
Open-access paper, CC-BY: **`10.1371/journal.pone.0260149`**
```python
from docling.document_converter import DocumentConverter
doc = DocumentConverter().convert("pone.0260149.pdf").document
print(doc.export_to_markdown())
```
### The same word, three times in one document
| where | how the PDF prints it | docling 2.124.0 |
|---|---|---|
| p1, line ends `…undecanal or n-`, next line begins `tridecanal` | broken at the line end | `undecanal or ntridecanal` :x: |
| p9, line ends `…n-dodecanal (77) and n-`, next line begins `tridecanal (84)` | broken at the line end | `and ntridecanal (84)` :x: |
| p11, mid-line | `n-octanal (10) and n-tridecanal (84) could be` | `and n-tridecanal (84)` :white_check_mark: |
**Docling's own output already contains the correct form.** The third row is preserved because the hyphen is not at a line break. The first two are destroyed because it is.
Line position alone cannot resolve these: the hyphen is line-final *and* part of the word. But the document says so plainly elsewhere:
- The p1 line reads `…n-octanal, n-nonanal, n-decanal, n-dodecanal, undecanal or n-` — four instances of the same construction, unbroken, on the very line that then breaks at `n-`.
- The p9 line reads `…n-undecanal (52), n-dodecanal (77) and n-`, likewise.
- `n-tridecanal` itself appears unbroken on p11, and Docling extracts it correctly there.
A dehyphenation pass that checked the forms already present in the document before deleting a hyphen would get all three right.
### Why this matters beyond the character count
The losses fall on chemical nomenclature, where the hyphen is part of the name: `n-decanal`, `4,5-dimethylthiazole`, `6-methyl-5-hepten-2-one`. `ndecanal` is not a variant spelling but a different string, and it breaks exact search and entity linking against any chemical vocabulary. The same applies to gene and compound identifiers.
For comparison, PyMuPDF preserves the hyphen in all three positions.
### Docling version
| version | p1 (line-final) | p9 (line-final) |
|---|---|---|
| 2.107.0 | `n- tridecanal` | `ntridecanal` |
| 2.117.0 | `n- tridecanal` | `ntridecanal` |
| **2.118.0** | **`ntridecanal`** | `ntridecanal` |
| 2.121.0 / 2.122.0 / 2.124.0 | `ntridecanal` | `ntridecanal` |
The p9 case merges in every version tested. The p1 case changed at **2.118.0**, whose release notes carry *"Dehyphenate hard continuations"*.
Tested docling 2.107.0, 2.117.0, 2.118.0, 2.121.0, 2.122.0, 2.124.0. `PyPdfiumDocumentBackend`. Identical results under torch 2.13.0+cpu and 2.14.0.
### Python version
Python 3.12.13, Linux
Contributor guide
Research direction
Start by reproducing the issue with DocumentConverter on pone.0260149.pdf and inspect the dehyphenation path used by PyPdfiumDocumentBackend before export_to_markdown(). Compare the line-final and mid-line n-tridecanal examples across the reported versions. Done means soft line breaks still join while genuine hyphens such as n-tridecanal remain searchable and intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100