docling-project / docling-project/docling-core

Serializers: `meta.language` renders as prose (Markdown/plain text print the pydantic repr, HTML shows a "Meta" block) and `HTMLParams.html_lang` is never emitted

Open
#745 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
282
Forks
214
Avg merge
2d 16h
Merged PRs (30d)
23

Description

## Summary

`BaseMeta.language` (`LanguageMetaField`, "the detected human language of the node content, expressed as a BCP 47 code") is the right slot for per-element language, and we now populate it downstream for PDF/UA `/Lang` tagging. But once it is set, every text serializer treats it as content:

1. **Markdown and plain text print the field's pydantic repr.** `MarkdownMetaSerializer._serialize_meta_field` has no `LanguageMetaField` branch, so it falls through to `str(field_val)` (`markdown.py:478` on main) and the document gains a paragraph like `confidence=0.99 created_by='langdetect' code=`. Plain text inherits the same serializer.
2. **HTML renders the language as visible text**, a `` block with `language: de`, instead of a `lang="de"` attribute on the element (WCAG 3.1.2 Language of Parts).
3. **`HTMLParams.html_lang` is declared (`html.py:111`) but never used**: the root element is the hard-coded string `""` (`html.py:1150`), so there is no way to set `` (WCAG 3.1.1 Language of Page) through the API.

Checked against v2.81.0 (installed) and `main` as of 2026-09-03 (v2.94.1); I found no existing issue or PR on these.

## Reproduction

```python
from docling_core.types.doc.base import BoundingBox, CoordOrigin, Size
from docling_core.types.doc.document import BaseMeta, DoclingDocument, LanguageMetaField, ProvenanceItem
from docling_core.types.doc.labels import DocItemLabel, HumanLanguageLabel
from docling_core.transforms.serializer.html import HTMLDocSerializer, HTMLParams

doc = DoclingDocument(name="demo")
doc.add_page(page_no=1, size=Size(width=612, height=792))
item = doc.add_text(
label=DocItemLabel.PARAGRAPH,
text="Dieser Absatz ist auf Deutsch.",
prov=ProvenanceItem(page_no=1, bbox=BoundingBox(l=0, t=100, r=100, b=0, coord_origin=CoordOrigin.BOTTOMLEFT), charspan=(0, 0)),
)
item.meta = BaseMeta(language=LanguageMetaField(code=HumanLanguageLabel.DE, confidence=0.99, created_by="langdetect"))

print(repr(doc.export_to_markdown()))
# "Dieser Absatz ist auf Deutsch.\n\nconfidence=0.99 created_by='langdetect' code="
print(repr(doc.export_to_text()))
# same repr appended as a paragraph

html = HTMLDocSerializer(doc=doc, params=HTMLParams(html_lang="de")).serialize().text
print("lang=" in html) # False — is emitted without the attribute
# body contains: …de
```

## Expected

- Markdown / plain text: a language code is metadata, not content. Either skip `LanguageMetaField` in `_serialize_meta_field` (return `None`) or, at most, emit it only under `mark_meta=True` as `[Language] de`. The current fallthrough to `str()` also means any future `BasePrediction`-derived field leaks its repr the same way, so a safer default for the `else` branch would be to skip unknown field types unless `mark_meta` is set.
- HTML: emit `lang=""` on the element that carries the text (`

`, `

`, ``, …) when `item.meta.language` differs from the document language, and drop the language row from the meta `` block (or keep it only when the language is the same as the document's, which is never useful).
- HTML root: honour `HTMLParams.html_lang` on ``. Reasonable default when `html_lang` is left at its default: the character-weighted majority of `meta.language` across the body, falling back to `"en"` as today.

## Why it matters

Accessible HTML output needs the page language on the root and the language of parts on the elements; screen readers switch pronunciation on these attributes. Today the only way to get clean text output once `meta.language` is populated is `blocked_meta_names={"language"}`, which callers have to know to pass on every serializer, and `` cannot be set at all without post-processing the string.

## Offer

Happy to open a PR with: the `LanguageMetaField` branch in the Markdown meta serializer (skip unless `mark_meta`), `lang` attributes in the HTML element serializers keyed off `meta.language`, the `` emission from `html_lang` with a majority-language default, and tests for the three cases above. Downstream context: YRA's yra-doc-pipeline populates `meta.language` from a deterministic detector and maps it to PDF/UA `/Lang`; the same data should reach the HTML export.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start with MarkdownMetaSerializer._serialize_meta_field at markdown.py:478 and HTMLParams/html.py:111, then inspect the root emission at html.py:1150 and reproduce the three cases in the issue. Done means language metadata no longer becomes text, HTML emits element and root lang attributes as specified, and tests cover Markdown, plain text, and HTML output.

Written by the indexing model from the issue text.

Assessment

Tech stack
html, markdown, python
Domain
accessibility, documentation
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.