docling-project / docling-project/docling
Control HTML document Unicode decoding
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Requested feature
Sometimes previous tools, e.g., OCR libraries output incorrectly encoded HTML. Because of visual similarity, for example, some undesired and incorrect character like https://www.compart.com/en/unicode/U+E157 is encoded, instead of https://www.compart.com/en/unicode/U+2630. Currently, when Docling parses an HTML document with such a character, it (or rather, BeautifulSoup) escapes these characters. For example, this heading item:
```html
Contents
```
ends up with the `.text` value:
```python
'Contents\ue157'
```
I have not found a straightforward way to control this behavior from within Docling or BeautifulSoup.
### Alternatives
I have not found a robust and direct method to process these escapes from within Python. String substitution tricks are possible but at a performance cost.
Contributor guide
Assessment
This issue has not been assessed yet.