RFC: MAIC ETL — document extraction and knowledge-source pipeline for OpenMAIC

## Summary

OpenMAIC currently treats uploaded course material as a PDF-centered path: upload a single PDF, parse it with a PDF provider, truncate the extracted text if needed, and pass `pdfText` / `pdfImages` into generation.

That model has served the first version well, but it is now limiting several related efforts:

- multi-format course material upload
- MinerU support beyond PDF
- multi-document course material bundles
- long textbook handling and chapter-aware preprocessing
- question bank / reference material support
- future RAG and knowledge-source workflows

This RFC proposes **MAIC ETL** as a restrained, incremental framework for these capabilities.

The goal is not to build a full knowledge-base product first. The goal is to generalize the existing PDF parsing path into a document extraction pipeline, using MinerU multi-format support as the first reference implementation.

## Motivation

Several current issues and PRs point to the same architectural pressure:

- #332 notes that the upload pipeline is hardcoded around PDF, and that PRs touching new formats or document processing conflict with each other.
- #611 adds TXT/DOCX/PDF parsing, but still needs a stable place to land that does not keep extending the old PDF-specific model.
- #41 and #140 request MD/TXT and DOCX support.
- #148 and #160 request multiple PDFs per generation.
- #52 and #65 request long-document summarization instead of simple truncation.
- #335 describes a real textbook use case where large illustrated documents need chapter slicing and image filtering.
- #43 requests question-bank reference material during generation.
- #258 already made web search query rewriting aware of uploaded PDF text, showing that downstream features need document context.
- #515 shows an external knowledge-graph-to-classroom workflow that could later fit into the same source framework.

These should not become a pile of unrelated feature patches. They should gradually form a small standard layer inside OpenMAIC.

## Proposed Direction

Introduce **MAIC ETL** as a staged roadmap:

```text
Source / File
-> Extractor Provider
-> DocumentArtifact
-> DocumentBundle
-> Transform
-> Index / Retrieve
-> Generation / Classroom QA
```

The first milestone should stay narrow:

```text
file -> document extractor provider -> DocumentArtifact
```

No vector database, no full RAG workspace, and no general knowledge-base UI in the first milestone.

## Milestone 0: Rename the Boundary

Generalize the current PDF-specific concepts into document-level concepts.

Examples:

- `PDFProvider` -> `DocumentExtractorProvider`
- `PDFParserConfig` -> `DocumentExtractorConfig`
- `ParsedPdfContent` -> `DocumentArtifact`
- `parse-pdf` -> `extract-document`
- `pdfFile` / `pdfText` / `pdfImages` -> document-oriented names

This can be done incrementally and should avoid database/storage migration unless necessary.

## Milestone 1: MinerU Multi-Format as Reference Provider

MinerU already supports more than PDF in its upstream ecosystem. OpenMAIC should use MinerU multi-format extraction as the first reference implementation for the generalized document extraction layer.

The provider should declare capabilities instead of relying on hardcoded assumptions:

```ts
DocumentExtractorProvider {
id: string
displayName: string
supportedMimeTypes: string[]
capabilities: {
text: boolean
images: boolean
tables: boolean
formulas: boolean
layout: boolean
ocr: boolean
async: boolean
}
}
```

The output should be normalized into a small OpenMAIC-owned artifact:

```ts
DocumentArtifact {
metadata
outline?
blocks
assets
citations?
diagnostics
providerRaw?
}
```

`providerRaw` keeps provider-specific data available without making MinerU's exact output the OpenMAIC standard.

## Milestone 2: Multi-Format Upload

Once the extractor boundary exists, the upload UI can move from "Upload PDF" to "Upload course material".

Initial scope:

- single file
- multiple formats
- provider selected by user or capability match
- clear diagnostics when a format/provider combination is unsupported

This should unblock or simplify #611, #41, and #140.

## Milestone 3: Document Bundles

After single-document multi-format extraction works, support multiple documents as a `DocumentBundle`.

This should address #148 / #160 without tying the design to PDF-only aggregation.

Key questions:

- document order
- per-document metadata
- text budget allocation
- image / vision budget allocation
- citations back to source document and page/block

## Milestone 4: Transform Layer

Only after extraction and bundles are stable, add transform steps:

- chapter detection
- long-document summarization
- noise removal
- image filtering
- table/formula normalization
- question-bank/reference-material shaping

This milestone connects to #52, #65, #335, and #43.

## Milestone 5: RAG Prototype

RAG should build on the previous layers, not replace them.

The first RAG scope should be OpenMAIC-specific:

- retrieval for course generation context
- retrieval for classroom QA context

Initial pieces:

- chunk policy
- embedding provider
- index backend
- retrieve policy
- context assembly with citations

This should remain separate from the first extraction milestone.

## Non-Goals for the First Milestone

- No full knowledge-base product UI.
- No vector database requirement.
- No persistent workspace requirement.
- No attempt to support every connector/provider at once.
- No claim that MinerU output is the OpenMAIC standard.
- No large rewrite of generation beyond the minimum needed to consume the generalized artifact.

## Suggested First PR

The first implementation PR could be:

**Generalize PDF parsing into document extraction, with MinerU multi-format as the first reference provider.**

Expected scope:

- introduce `DocumentExtractorProvider`
- introduce `DocumentArtifact`
- keep compatibility with current PDF parsing behavior
- add capability metadata for current providers
- route current PDF parsing through the new abstraction
- add MinerU multi-format support where the provider already supports it
- preserve existing generation behavior as much as possible

This gives the project a concrete implementation while letting the standard emerge from real provider behavior.

Start by reviewing the current PDF parsing path and the related issues #332 and #611 to understand the existing boundary and compatibility constraints. The proposed first PR is done when document extraction is generalized around DocumentExtractorProvider and DocumentArtifact, current PDF behavior remains compatible, provider capabilities are represented, and MinerU multi-format support is routed through the new abstraction.

Written by the indexing model from the issue text.