docling-project / docling-project/docling-parse

word separation ignores the font's own space width, so tightly kerned text loses word boundaries

Open
#338 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
333
Forks
80
Avg merge
1d 14h
Merged PRs (30d)
10

Description

moved here from docling-project/docling#4078 at @cau-git's request. originally reported
as docling#3884.

## what happens

when a pdf draws text with a `TJ` array whose words are separated only by kerning
numbers, no space glyph exists in the content stream and word boundaries have to be
inferred from geometry. past a certain tightness they are lost entirely:

```
'Causeofdeathperlegaloutcomefornon-humancasesintheNetherlands'
```

expected:

```
'Cause of death per legal outcome for non-human cases in the Netherlands'
```

it is threshold dependent. at kern -190 only the first pair fuses, at -170 the whole
line fuses into a single word cell.

## root cause

`page_item::merge_with` in `src/parse/page_items/page_cell.h` decides it in
one comparison:

```cpp
double d0 = std::sqrt((r_x1-other.r_x0)*(r_x1-other.r_x0) + (r_y1-other.r_y0)*(r_y1-other.r_y0));
if(delta`. so the test is "is this gap
wider than a third of the average glyph in the preceding cell", a proxy for the width of
a space rather than the width of a space.

the proxy is what fails. `average_char_width()` is the mean advance over the cell's own
glyphs, so a cell of narrow letters lowers the bar and a cell of wide ones raises it,
independently of how wide that font actually sets a space.

## the part that makes this straightforward

the parser already computes the right quantity, and already carries it on every cell.
`src/parse/pdf_states/text.h:584`:

```cpp
double space_width=0;
{
double w0 = font.get_space_width();
double w1 = (w0 / 1000.0 * font_size * h_scaling);
std::array rect = compute_rect(font_descent, font_ascent, w1);
space_width = std::sqrt((rect[2]-rect[0])*(rect[2]-rect[0])+
(rect[3]-rect[1])*(rect[3]-rect[1]));
}
```

that is the font's declared space advance, scaled by font size and horizontal scaling and
carried through `compute_rect` so it is correct under rotation. it is assigned to
`cell.space_width` at `text.h:702`, serialised at `page_cell.h:204`, and then never
consulted by the merge logic.

## what i am going to do

scale the threshold by the cell's own `space_width` when it is populated, and keep the
current `average_char_width()` expression as the fallback for when it is not.
`pdf_decoders/page.h:1425` sets it to zero on at least one path, so the fallback is
required rather than defensive.

this keeps the existing factors meaningful and their defaults untouched, so a document
whose spacing is judged correctly today continues to be judged the same way. it only
changes what the factor is a fraction *of*, from a proxy to the real measurement.

it also lines up with the reasoning already recorded in
`page_item_sanitators/cells.h`, that an explicit space glyph is more reliable than the
geometric gap heuristic "which is ambiguous for tightly-set fonts with narrow spaces".
this removes that ambiguity for the case where the glyph is absent, by asking the font
how wide its space is instead of guessing from neighbouring glyphs.

pr to follow, with unit tests built through `tests/pdf_builder.py` and the regression
corpus run to show what moves.

Contributor guide

Open the contributing guide

Research direction

Start in src/parse/page_items/page_cell.h at page_item::merge_with, then trace cell.space_width from src/parse/pdf_states/text.h:584 and its assignment at text.h:702; check the zero-value path in pdf_decoders/page.h:1425. Build the regression cases through tests/pdf_builder.py and run the regression corpus. Done means tightly kerned text preserves the expected word boundaries, unset space widths still use the existing fallback, and default factors remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.