Investigation: Off-the-shelf PDF Parsing / MCP Services vs AKD-Core Pipeline

Is your feature request related to a problem?

Our current AKD-Core PDF parsing pipeline is functional and integrated into the system. As AKD grows and document complexity increases, it is worth evaluating whether off-the-shelf PDF resolving/parsing services (including MCP-based options) could provide measurable improvements in extraction quality, token efficiency, or operational scalability.

This is a proactive architectural evaluation rather than a reaction to failure.

Describe the solution you'd like

Perform a structured comparison between:

• The existing AKD-Core PDF pipeline
• Off-the-shelf PDF parsing / resolving services (including MCP-based options)

The goal is to quantify tradeoffs across:

• Extraction fidelity (scientific formatting, tables, references, multi-column layouts)
• Token footprint and downstream LLM context impact
• Integration complexity (SDK maturity, infrastructure changes, authentication)
• Latency and cost per document
• Observability compatibility
• Long-term scalability characteristics

Deliverable: A comparison matrix with quantified metrics and a recommendation (retain in-house, wrap MCP, or adopt an external solution).

Describe alternatives you've considered

• Continue iterating on and optimizing the existing AKD-Core PDF pipeline.

Additional context

This decision impacts:

• Token efficiency in downstream agent workflows
• Parsing robustness for research-grade PDFs
• Long-term architectural simplicity
• Operational scalability

Representative AKD PDFs will be used for benchmarking to ensure realistic evaluation.