docling-project / docling-project/docling

Control HTML document Unicode decoding

Open
#682 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Requested feature

Sometimes previous tools, e.g., OCR libraries output incorrectly encoded HTML. Because of visual similarity, for example, some undesired and incorrect character like https://www.compart.com/en/unicode/U+E157 is encoded, instead of https://www.compart.com/en/unicode/U+2630. Currently, when Docling parses an HTML document with such a character, it (or rather, BeautifulSoup) escapes these characters. For example, this heading item:

```html

Contents


```

ends up with the `.text` value:

```python
'Contents\ue157'
```

I have not found a straightforward way to control this behavior from within Docling or BeautifulSoup.

### Alternatives

I have not found a robust and direct method to process these escapes from within Python. String substitution tricks are possible but at a performance cost.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.