docling-project / docling-project/docling
Further improvements for HTML backend
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
Here are some thoughts on further improvements in the HTML backend as we go forward:
### HTML Titles as metadata instead of an element.
Currently HTML Titles ( tag in the <head> of the HTML page) is parsed into title element of Docling Document. This leads to an extra document element that is not visible on the actual page when rendered. Would be great to move it into metadata of a Docling document instead. This would change behavior, but on my opinion for the better.
[doc.add_title](https://github.com/docling-project/docling/blob/e36125ba2ddfbe584fc752e6dc7ca0f0f8f58d87/docling/backend/html_backend.py#L472)
### Improve DOM Tree distance heuristic for key-value extraction from HTML page
Currently we have "hardcoded" certain depth/distance metrics in our backend logic, that ensures somewhat local positioning of key and value on the page (think forms). Perhaps there could be a better way to do it.
[if descendant_value_count > 3](https://github.com/docling-project/docling/blob/e36125ba2ddfbe584fc752e6dc7ca0f0f8f58d87/docling/backend/html_backend.py#L3612)
[if len(value_tags) > 8 and len(key_text) > 120:](https://github.com/docling-project/docling/blob/e36125ba2ddfbe584fc752e6dc7ca0f0f8f58d87/docling/backend/html_backend.py#L3620)
### Heuristic for long-page pagination
Some web-pages can be very long, we also can render them into an image and materialize provenances with specific coordinates from the image. Would be great to design heuristic for automatic pagination, that would cut long HTMLs into "virtual pages", and produce multi-page docling document instead of one very long page.
Contributor guide
Assessment
This issue has not been assessed yet.