docling-project / docling-project/docling
Integrate Docling in Elasticsearch
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Requested feature
#### Background
[Docling](https://github.com/DS4SD/docling) reads popular document formats (PDF, DOCX, PPTX, XLSX, Images, HTML, AsciiDoc & Markdown), converts them in a unified data model with rich document representation ( `DoclingDocument` class in [docling-core](https://github.com/DS4SD/docling-core)) and exports to Markdown and JSON.
[Elasticsearch](https://www.elastic.co/elasticsearch) is an open source distributed, RESTful search and analytics engine, scalable data store, and vector database. Among its features, the [Attachment Processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/attachment.html) lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by using the Apache text extraction library [Tika](https://tika.apache.org/)
#### Goal
This feature request consists of designing and implementing an integration solution of Docling with Elasticsearch to enable the parsing and indexing of document content in common formats (PDF, HTML, MS Office, ...).
Some aspects will need to be addressed, including:
- Running Docling on the JVM, since Docling is written in Python and Elasticsearch in Java. An existing option, already in place in Elasticsearch, is leveraging [Jython](https://www.jython.org/).
- Deciding the integration options: new implementation of [Attachment Processor](https://github.com/elastic/elasticsearch/blob/main/modules/ingest-attachment/src/main/java/org/elasticsearch/ingest/attachment/AttachmentProcessor.java), an Elasticsearch plugin, ...
- The indexing options:
- text only: export the resulting Docling object to Markdown and index it as a `text` type field, as it is done with the `content` field in the Attachment Processor
- document structure: export the resulting Docling object to JSON and allow users to select the fields to extract and index (such as paragraphs, tables, ...).
### Alternatives
- Do not provide a native integration, instead provide a tool in Docling to create an index with custom mappings to store and index Docling documents exported as JSON. This approach was implemented in the [document legacy version](https://github.com/DS4SD/docling-core/blob/main/docling_core/types/legacy_doc/document.py).
- Explore other open-source, enterprise search suites, like [OpenSearch](https://aws.amazon.com/what-is/opensearch/)
Contributor guide
Assessment
This issue has not been assessed yet.