docling-project / docling-project/docling

Prefer native structured math (MathML/OMML) over formula reconstruction

Open
#4,081 2 comments 0 reactions 2 assignees Claimed by @DanielNg0729 View on GitHub
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

## Feature request: extract embedded MathML / native structured equations before formula reconstruction

### Description

Does Docling currently detect and extract a structured mathematical representation that is already embedded in the source document, instead of reconstructing the formula visually using the formula enrichment model?

In particular, modern accessible/tagged PDFs can contain a MathML representation associated with a `Formula` structure element. Depending on the producer, this may be represented through mechanisms such as:

- an associated file / embedded XML stream containing MathML;
- MathML-related structure elements in the PDF structure tree;
- producer-specific attributes containing MathML (for example, Microsoft documents PDF/UA output containing an `MSFT_MathML` attribute).

If this information is available in the PDF, it seems preferable to extract it directly rather than:

1. detecting the rendered formula;
2. interpreting the visual glyphs;
3. reconstructing a LaTeX representation with the formula enrichment model.

The embedded representation should generally be more faithful to the author's original mathematical structure.

### Current behavior

The Docling documentation describes formula understanding as analyzing equation formulas and extracting their LaTeX representation.

As far as I can tell, it is not clear whether Docling first checks the PDF structure tree / associated files for an existing MathML representation.

Could you clarify whether this is currently supported?

### Proposed behavior

When parsing a PDF, Docling could use the following priority:

1. **Native embedded MathML**, when explicitly associated with the formula in the PDF.
2. Other structured mathematical metadata exposed by the PDF producer.
3. Existing formula understanding / visual reconstruction as a fallback.

For example, conceptually:

```text
PDF Formula structure element
|
+-- MathML available?
| |
| +-- yes -> parse and preserve MathML
|
+-- no -> existing formula enrichment
|
+-- reconstructed LaTeX
```

Ideally the resulting `FormulaItem` could retain both representations when available, e.g. something conceptually similar to:

```python
FormulaItem(
text="...",
latex=r"\frac{x}{y}",
mathml="",
)
```

The precise data model is of course up to Docling.

### Why this would be useful

This would avoid unnecessary lossy formula recognition for born-digital accessible documents.

Potential advantages:

- higher fidelity;
- no dependence on formula-recognition model accuracy;
- better handling of complex notation;
- preservation of accessibility information;
- easier conversion to HTML;
- better downstream use in scientific-document pipelines;
- possibility to preserve both MathML and LaTeX representations.

It could also substantially improve extraction from documents produced specifically with accessible-math workflows such as modern tagged LaTeX/PDF.

### DOCX

There appears to be an analogous opportunity for DOCX, although the underlying representation is different.

Word equations in `.docx` are normally stored as **OMML (Office Math Markup Language)** rather than MathML.

Docling already appears to parse OMML and convert it to LaTeX. Would it make sense for Docling to also preserve the native structured representation, and/or expose a MathML conversion alongside the generated LaTeX?

Conceptually the desired behavior could therefore be:

```text
PDF
embedded MathML -> FormulaItem.mathml
-> optionally derive LaTeX

DOCX
native OMML -> preserve structured math
-> MathML
-> LaTeX
```

rather than converting every source immediately into only LaTeX.

Microsoft documents an OMML → MathML mapping, so MathML could potentially provide a common normalized representation across PDF and DOCX inputs.

### Suggested questions

1. Does the PDF backend currently inspect tagged-PDF `Formula` elements for associated MathML?
2. Does it inspect PDF associated files / embedded XML associated with formulas?
3. Does it recognize producer-specific MathML attributes such as `MSFT_MathML`?
4. If MathML is present, can it be exposed in `FormulaItem` rather than running formula enrichment?
5. For DOCX, could the existing OMML parser expose MathML in addition to LaTeX?
6. Would Docling be interested in using MathML as a common structured representation for native equations across document formats?

### References

- LaTeX Tagged PDF project: accessible mathematics can associate MathML with `Formula` structure elements.
- PDF Association / Well-Tagged PDF guidance: MathML may be represented using associated files and/or MathML structure elements.
- Microsoft 365 MathML documentation: Office Math/OMML can be exported to MathML, and PDF/UA export can expose MathML for equations.
- Docling formula enrichment documentation: currently describes extraction of a LaTeX representation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.