boostorg / boostorg/website-v2

Incorrect handling of HTML5 named character references in Boost.Bloom documentation

Open
#1,851 2 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
HTML
Stars
18
Forks
28
Avg merge
2d 12h
Merged PRs (30d)
77

Description

Boost.Bloom uses HTML named character references in several places, for instance in [the acknowledgement section](https://github.com/boostorg/bloom/blob/develop/doc/bloom/acknowledgements.adoc?plain=1), where we find:

```
Rubén Pérez,
```

and

```
Andrzej Krzemieński,
```

which should be displayed as **Rubén Pérez** and **Andrzej Krzemieński**, respectively. Yet, we find that the website [displays the latter incorrectly](https://www.boost.org/doc/libs/develop/libs/bloom/doc/html/bloom.html#acknowledgements):

Image

### Analysis of the problem

The HTML directly generated by Asciidoc for Boost.Bloom documentation looks like (for the portion of interest):

```

The Boost acceptance review took place between the 13th and 22nd of May,
2025. Big thanks to Arnaud Becheler for his expert managing. The
following people participated in the review:
Dmitry Arkhipov,
David Bien,
Claudio DeSouza,
Peter Dimov,
Vinnie Falco,
Alexander Grund,
Seth Heeren,
Andrzej Krzemieński,
Ivan Matek,
Christian Mazakas,
Rubén Pérez,
Kostas Savvidis,
Peter Turcan,
Tomer Vromen. Many thanks to all of them for their very helpful feedback.


```

(note the named character references **ń** and **é**), while the HTML after postprocessing by the website looks like:

```

The Boost acceptance review took place between the 13th and 22nd of May,
2025. Big thanks to Arnaud Becheler for his expert managing. The
following people participated in the review:
Dmitry Arkhipov,
David Bien,
Claudio DeSouza,
Peter Dimov,
Vinnie Falco,
Alexander Grund,
Seth Heeren,
Andrzej Krzemieński,
Ivan Matek,
Christian Mazakas,
Rubén Pérez,
Kostas Savvidis,
Peter Turcan,
Tomer Vromen. Many thanks to all of them for their very helpful feedback.


```

where **é** has been transformed into **é**, whereas **ń** has been transformed into **ń** (displayed by the browser as **ń**). So, the root cause for the problem is that

* The backend is parsing the generated HTML and replacing HTML named character references with their Unicode/UTF-8 equivalents.
* The backend fails to recognize **ń** as a valid named character reference and instead parses it as **&** followed by **nacute;**, which then proceeds to transform (presumably at a later rendering stage) into **&** followed by **nacute;**.

An important fact is that while **é** exists in HTML from as early as HTML 2.0, **ń** was only included in [HTML5/WHATWG](https://html.spec.whatwg.org/multipage/named-characters.html). HTML parsing seems to be done here:

https://github.com/boostorg/website-v2/blob/c62bad1f2db2aaaec2bb7667e07041a2bda55980/core/htmlhelper.py#L246

where `html` is provided by the [`lxml`](https://pypi.org/project/lxml/) module. This facility [does not support HTML5 named character references](https://onlinegdb.com/SCFMho5Ln):

```python
from lxml import html

root = html.fromstring("é")
content = html.tostring(root, encoding="unicode", method="html")
print(content)

root = html.fromstring("ń")
content = html.tostring(root, encoding="unicode", method="html")
print(content)
```
Output:
```

é


ń


```
But the provided [html5parser facility does](https://onlinegdb.com/3-oK77EVy):

```python
from lxml import html
from lxml.html import html5parser

root = html5parser.fromstring("é")
content = html.tostring(root, encoding="unicode", method="html")
print(content)

root = html5parser.fromstring("ń")
content = html.tostring(root, encoding="unicode", method="html")
print(content)
```
Output:
```
é
ń
```
Another option could be to not translate HTML entities at all or to render them back as HTML entities rather than Unicode/UTF-8 characters.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.