docling-project / docling-project/docling
Extend `sanitize_text` with known exceptions
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
With the new reading order module we are currently missing some text normalization which were applied by the glm component. For example in https://github.com/DS4SD/deepsearch-glm/blob/main/src/andromeda/tooling/structs/normalizers/text_normalizer.h#L52.
Currently some text normalization is done in https://github.com/DS4SD/docling/blob/main/docling/models/page_assemble_model.py#L55, however
1. it is not complete (see complete glm tooling linked above)
2. it does not preserve the original text in the orig field
Some extra notes:
- while porting the normalization we should look at issues like this https://github.com/DS4SD/docling/issues/928.
- the text normalization should be in the `text` field, leaving the `orig` content with the original text.
Contributor guide
Assessment
This issue has not been assessed yet.