python / python/cpython

Handling of surrogates on JSON encoding

Open
#93,508 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

docs
Dominant language
Python
Stars
77.2k
Forks
36k
Avg merge
1d 9h
Merged PRs (30d)
558

Description

When json.dumps is called with ensure_ascii = True (the default), the output does not distinguish between Unicode characters outside of the BMP and their corresponding surrogate characters. This means that json.loads(json.dumps(s)) is not guaranteed to return s:

>>> json.loads(json.dumps('𝄞')) == '𝄞'
True
>>> json.loads(json.dumps('\ud834\udd1e')) == '\ud834\udd1e'
False

The only way to get surrogate characters back is to have them in the JSON string as actual characters, not escaped:

>>> json.loads( '"\ud83d\udca9"')  # Actual surrogate characters
'\ud83d\udca9'
>>> json.loads(r'"\ud83d\udca9"')  # Escaped surrogates
'💩'

Python does handle the case of lone surrogates reasonably (although RFC 8259 basically declares this undefined behaviour), so the round-trip does work in some cases:

>>> json.loads(json.dumps('\ud83d'))
'\ud83d'

It would seem reasonable to expect that json.loads(json.dumps(s)) == s would hold for any string. This isn't really a bug but rather a limitation of pure-ASCII encoding. However, I think it does deserve a mention in the documentation. Possibly at the end of the Character encodings section? I'm not quite sure.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the JSON documentation's Character encodings section, linked in the issue. Review the demonstrated behavior for non-BMP characters, surrogate pairs, and lone surrogates, then document the round-trip limitation and its relationship to ensure_ascii=True. Done means the section clearly explains when json.loads(json.dumps(s)) may not equal s.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.