docling-project / docling-project/docling

Integrate Docling in Elasticsearch

Open
#465 3 comments 1 reaction 1 assignee Claimed by @ceberam View on GitHub
enhancement
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Requested feature

#### Background

[Docling](https://github.com/DS4SD/docling) reads popular document formats (PDF, DOCX, PPTX, XLSX, Images, HTML, AsciiDoc & Markdown), converts them in a unified data model with rich document representation ( `DoclingDocument` class in [docling-core](https://github.com/DS4SD/docling-core)) and exports to Markdown and JSON.
[Elasticsearch](https://www.elastic.co/elasticsearch) is an open source distributed, RESTful search and analytics engine, scalable data store, and vector database. Among its features, the [Attachment Processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/attachment.html) lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by using the Apache text extraction library [Tika](https://tika.apache.org/)

#### Goal

This feature request consists of designing and implementing an integration solution of Docling with Elasticsearch to enable the parsing and indexing of document content in common formats (PDF, HTML, MS Office, ...).

Some aspects will need to be addressed, including:
- Running Docling on the JVM, since Docling is written in Python and Elasticsearch in Java. An existing option, already in place in Elasticsearch, is leveraging [Jython](https://www.jython.org/).
- Deciding the integration options: new implementation of [Attachment Processor](https://github.com/elastic/elasticsearch/blob/main/modules/ingest-attachment/src/main/java/org/elasticsearch/ingest/attachment/AttachmentProcessor.java), an Elasticsearch plugin, ...
- The indexing options:
- text only: export the resulting Docling object to Markdown and index it as a `text` type field, as it is done with the `content` field in the Attachment Processor
- document structure: export the resulting Docling object to JSON and allow users to select the fields to extract and index (such as paragraphs, tables, ...).

### Alternatives

- Do not provide a native integration, instead provide a tool in Docling to create an index with custom mappings to store and index Docling documents exported as JSON. This approach was implemented in the [document legacy version](https://github.com/DS4SD/docling-core/blob/main/docling_core/types/legacy_doc/document.py).
- Explore other open-source, enterprise search suites, like [OpenSearch](https://aws.amazon.com/what-is/opensearch/)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.