docling-project / docling-project/docling

Extend `sanitize_text` with known exceptions

Open
#1,127 1 comment 0 reactions 4 assignees Claimed by @vagenas View on GitHub
enhancement triage/close-stale
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

With the new reading order module we are currently missing some text normalization which were applied by the glm component. For example in https://github.com/DS4SD/deepsearch-glm/blob/main/src/andromeda/tooling/structs/normalizers/text_normalizer.h#L52.

Currently some text normalization is done in https://github.com/DS4SD/docling/blob/main/docling/models/page_assemble_model.py#L55, however
1. it is not complete (see complete glm tooling linked above)
2. it does not preserve the original text in the orig field

Some extra notes:
- while porting the normalization we should look at issues like this https://github.com/DS4SD/docling/issues/928.
- the text normalization should be in the `text` field, leaving the `orig` content with the original text.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.