apache / apache/seatunnel

[Feature][Connector-V2][RAG] Add optional anydoc backend for document-to-Markdown parsing

Open
#11,801 4 comments 0 reactions 1 assignee Claimed by @goutamadwant View on GitHub
help wanted
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [x] I searched existing issues and found related document/RAG work, but no issue that specifically discusses an `anydoc`-backed document-to-Markdown parser backend.

Related issues:

- #10889
- #9861
- #10608
- #9716
- #10916
- #10920

### Description

SeaTunnel already has an active Knowledge Sync / RAG direction, and the current `dev` branch has document-oriented file source support:

- `FileFormat` includes `MARKDOWN` and `PDF` read strategies.
- `PdfReadStrategy` extracts PDF document elements through PDFBox and can append RAG metadata fields.
- `MarkdownReadStrategy` parses Markdown into document elements and can append RAG metadata fields.
- `markdown_rag_metadata_enabled` and `pdf_rag_metadata_enabled` already expose the RAG metadata switch at file-source option level.

This issue proposes adding an optional `anydoc`-backed parser backend for document-to-Markdown conversion, then reusing the existing Markdown/document element path where possible.

`anydoc` is a Rust library from Firecrawl that converts Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and text-based PDF files to GitHub-Flavored Markdown. It currently publishes Node.js, Python, Rust, and WebAssembly bindings, and its repository is MIT licensed:

https://github.com/firecrawl/anydoc

Why this may fit SeaTunnel:

1. SeaTunnel's current PDF support is PDF-specific and PDFBox-based. It is useful, but it does not provide one shared document-to-Markdown path for Office documents, RTF, EPUB, CSV, and PDF.
2. SeaTunnel already has a Markdown reader that can turn Markdown into structured rows, so a document-to-Markdown parser backend can avoid duplicating structure extraction logic per document format.
3. Knowledge Sync needs stable document/chunk metadata and a predictable text representation before chunking, embedding, and vector sink lifecycle handling. Markdown is a reasonable intermediate contract for that layer.

### Suggested Scope

Add an optional document parser backend, for example:

```hocon
source {
LocalFile {
path = "/path/to/documents"
file_format_type = "document"
document_parser = "anydoc"
document_output_format = "markdown"
document_rag_metadata_enabled = true
}
}
```

Possible implementation shape:

1. Add a new `DOCUMENT` file format or a dedicated document parser option instead of overloading the existing `PDF` behavior.
2. Let the parser convert supported documents to Markdown first.
3. Reuse or share the Markdown document-element parsing path to emit rows with fields such as `element_id`, `element_type`, `heading_level`, `text`, `source_uri`, `document_id`, `chunk_id`, `chunk_index`, and `content_hash`.
4. Keep existing `file_format_type = pdf` behavior unchanged for compatibility.
5. Gate anydoc behind an explicit option so deployments do not take a native/external runtime dependency unless they opt in.

### Integration Concerns

This should be treated as an integration proposal, not as a simple Maven dependency bump:

- SeaTunnel is JVM/Maven based, while anydoc currently exposes Rust/Node/Python/WASM bindings rather than a normal Java artifact.
- A PoC could use an external command adapter, but production support should define a clear runtime packaging story, failure semantics, concurrency model, and cross-platform support.
- Scanned or image-only PDFs still require OCR. anydoc handles text-based PDFs locally but does not replace OCR.
- Existing PDFBox-based `PdfReadStrategy` should remain available unless the community explicitly decides otherwise.

### Acceptance Criteria

- A design is documented for where the parser backend belongs: file source format, transform, or shared document parser SPI.
- Existing PDF and Markdown behavior remains backward compatible.
- The supported format list is explicit, including the no-OCR limitation for scanned PDFs.
- The emitted schema is aligned with current Markdown/PDF RAG metadata and the Knowledge Sync metadata contract.
- Runtime dependency packaging is documented and tested for at least Linux and macOS, with clear behavior when the backend is unavailable.
- Unit tests cover text-based PDF, DOCX or PPTX, Markdown handoff, unsupported/scanned PDF behavior, and error handling.
- At least one e2e example shows document files being read, converted to Markdown/structured rows, chunked, and written to a downstream sink.

### Usage Scenario

Enterprise RAG and knowledge indexing pipelines often receive mixed document sets in object storage or local/HDFS paths:

- PDFs
- DOC/DOCX
- PPT/PPTX
- XLS/XLSX
- ODT/ODS/ODP
- RTF
- EPUB

A SeaTunnel pipeline should be able to discover those documents, normalize them into Markdown or document elements, add stable document/chunk metadata, then continue through chunking, embedding, and vector sink lifecycle handling.

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.