Not all HTML Entities are unescaped from title and other metadata (when double-escaped by websites)

Open
#820 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
javascript
Domain
web-dev

Research direction

Start in Readability.js at the linked unescaping function and the metadata.title call to _unescapeHtmlEntities. Reproduce the Scientific American example, then verify that double-escaped entities such as ’ are decoded in the extracted title and related metadata without unsafe parsing.

Written by the indexing model from the issue text.

Description

has-website-testcase metadata

There is a function in the code for unescaping HTML entities:

https://github.com/mozilla/readability/blob/2524fe371da2356b0bb79e0d34b028fa23388cd3/Readability.js#L1353-L1365

However, it does not capture all possible HTML entities (’)

Example

On the page https://www.scientificamerican.com/podcast/episode/heres-why-actors-are-so-worried-about-ai/

There is this meta tag: <meta property="og:title" content="Here&amp;rsquo;s Why Actors Are So Worried about AI">

The page title is extracted from it.

Special html entities are supposed to be unescaped by this function, but they are not:

https://github.com/mozilla/readability/blob/2524fe371da2356b0bb79e0d34b028fa23388cd3/Readability.js#L1553

The metadata.title before calling this._unescapeHtmlEntities and after is the same:

Here&rsquo;s Why Actors Are So Worried about AI

Solution & Workaround

According to https://stackoverflow.com/a/34064434/8584605, a more effective (and still safe) way to unescape HTML entities would look like this:

function htmlDecode(input) {
  const doc = new DOMParser().parseFromString(input, "text/html");
  return doc.documentElement.textContent;
}

Until this bug is fixed, I would be using the above function to post-process the title outputted by Readability.js

Dominant language
JavaScript
Stars
11.5k
Forks
732
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from mozilla/readability

All issues in mozilla/readability

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.