Handling of surrogates on JSON encoding
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 36k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 558
Description
When json.dumps is called with ensure_ascii = True (the default), the output does not distinguish between Unicode characters outside of the BMP and their corresponding surrogate characters. This means that json.loads(json.dumps(s)) is not guaranteed to return s:
>>> json.loads(json.dumps('𝄞')) == '𝄞'
True
>>> json.loads(json.dumps('\ud834\udd1e')) == '\ud834\udd1e'
False
The only way to get surrogate characters back is to have them in the JSON string as actual characters, not escaped:
>>> json.loads( '"\ud83d\udca9"') # Actual surrogate characters
'\ud83d\udca9'
>>> json.loads(r'"\ud83d\udca9"') # Escaped surrogates
'💩'
Python does handle the case of lone surrogates reasonably (although RFC 8259 basically declares this undefined behaviour), so the round-trip does work in some cases:
>>> json.loads(json.dumps('\ud83d'))
'\ud83d'
It would seem reasonable to expect that json.loads(json.dumps(s)) == s would hold for any string. This isn't really a bug but rather a limitation of pure-ASCII encoding. However, I think it does deserve a mention in the documentation. Possibly at the end of the Character encodings section? I'm not quite sure.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the JSON documentation's Character encodings section, linked in the issue. Review the demonstrated behavior for non-BMP characters, surrogate pairs, and lone surrogates, then document the round-trip limitation and its relationship to ensure_ascii=True. Done means the section clearly explains when json.loads(json.dumps(s)) may not equal s.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100