docling-project / docling-project/docling

Dehyphenation merges line-final hyphens that belong to the word, losing them irrecoverably (regression in 2.118.0)

Open
#4,151 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug

When a hyphenated word is broken across a line, Docling joins the halves and deletes the hyphen without checking whether the hyphen is part of the word. n-tridecanal becomes ntridecanal.

Mid-line hyphens are handled correctly, so the logic is already position-aware. The gap is that a line-final hyphen can also be a real hyphen.

The loss is unrecoverable: once the hyphen is gone, nothing downstream can tell whether the page printed ntridecanal or n-tridecanal. The opposite failure — leaving n- tridecanal unjoined, which 2.107.0 does — is wrong but repairable by anyone later.

### Steps to reproduce

Open-access paper, CC-BY: **`10.1371/journal.pone.0260149`**

```python
from docling.document_converter import DocumentConverter
doc = DocumentConverter().convert("pone.0260149.pdf").document
print(doc.export_to_markdown())
```

### The same word, three times in one document

| where | how the PDF prints it | docling 2.124.0 |
|---|---|---|
| p1, line ends `…undecanal or n-`, next line begins `tridecanal` | broken at the line end | `undecanal or ntridecanal` :x: |
| p9, line ends `…n-dodecanal (77) and n-`, next line begins `tridecanal (84)` | broken at the line end | `and ntridecanal (84)` :x: |
| p11, mid-line | `n-octanal (10) and n-tridecanal (84) could be` | `and n-tridecanal (84)` :white_check_mark: |

**Docling's own output already contains the correct form.** The third row is preserved because the hyphen is not at a line break. The first two are destroyed because it is.

Line position alone cannot resolve these: the hyphen is line-final *and* part of the word. But the document says so plainly elsewhere:

- The p1 line reads `…n-octanal, n-nonanal, n-decanal, n-dodecanal, undecanal or n-` — four instances of the same construction, unbroken, on the very line that then breaks at `n-`.
- The p9 line reads `…n-undecanal (52), n-dodecanal (77) and n-`, likewise.
- `n-tridecanal` itself appears unbroken on p11, and Docling extracts it correctly there.

A dehyphenation pass that checked the forms already present in the document before deleting a hyphen would get all three right.

### Why this matters beyond the character count

The losses fall on chemical nomenclature, where the hyphen is part of the name: `n-decanal`, `4,5-dimethylthiazole`, `6-methyl-5-hepten-2-one`. `ndecanal` is not a variant spelling but a different string, and it breaks exact search and entity linking against any chemical vocabulary. The same applies to gene and compound identifiers.

For comparison, PyMuPDF preserves the hyphen in all three positions.

### Docling version

| version | p1 (line-final) | p9 (line-final) |
|---|---|---|
| 2.107.0 | `n- tridecanal` | `ntridecanal` |
| 2.117.0 | `n- tridecanal` | `ntridecanal` |
| **2.118.0** | **`ntridecanal`** | `ntridecanal` |
| 2.121.0 / 2.122.0 / 2.124.0 | `ntridecanal` | `ntridecanal` |

The p9 case merges in every version tested. The p1 case changed at **2.118.0**, whose release notes carry *"Dehyphenate hard continuations"*.

Tested docling 2.107.0, 2.117.0, 2.118.0, 2.121.0, 2.122.0, 2.124.0. `PyPdfiumDocumentBackend`. Identical results under torch 2.13.0+cpu and 2.14.0.

### Python version

Python 3.12.13, Linux

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with DocumentConverter on pone.0260149.pdf and inspect the dehyphenation path used by PyPdfiumDocumentBackend before export_to_markdown(). Compare the line-final and mid-line n-tridecanal examples across the reported versions. Done means soft line breaks still join while genuine hyphens such as n-tridecanal remain searchable and intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.