FasterXML / FasterXML/jackson-core

Consider validating raw unpaired UTF-16 surrogates in ReaderBasedJsonParser

Open
#1,705 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
2.4k
Forks
928
Avg merge
2d 18h
Merged PRs (30d)
24

Description

`ReaderBasedJsonParser` only checks surrogate pairing inside `\uXXXX` escapes (#1684). Raw UTF-16
code units that come in through a `Reader` are copied straight through, in both string values and
field names. Same on 2.22.2 and 3.1.4:

```
kind now proposed
U+D800 lone high accept reject
U+DC00 lone low accept reject
U+DC00 + U+D800 reversed accept reject
U+D800 + U+DC00 valid pair accept accept
```

```java
// both accept; the value/name comes back as a 1-char String holding 0xd800
new JsonFactory().createParser(new StringReader("{\"k\":\"\uD800\"}")); // string value
new JsonFactory().createParser(new StringReader("{\"\uD800\":1}")); // field name
```

[RFC 7493 section 2.1](https://www.rfc-editor.org/rfc/rfc7493.html#section-2.1) (I-JSON) is explicit
that the prohibition "applies both to characters encoded directly in UTF-8 and to those which are
escaped". Plain JSON ([RFC 8259 section 8.2](https://www.rfc-editor.org/rfc/rfc8259.html#section-8.2))
only calls the result unpredictable, so this is a tightening rather than a compliance bug.

Distinct from #363, which rejects ill-formed *UTF-8*: the same code points fail on the byte path and
pass on the `Reader` path, so the inconsistency is across input types rather than a gap in #363.

This is a different code path from #1684 and a much bigger compatibility question, so I have kept it
out of that PR. It does not block it.

**What is clear**

Lone high, lone low and reversed pairs are not valid UTF-16, so they should be rejected. A valid
pair has to keep working.

**Needs input**

Whether this goes in as opt-in on 2.x, or as the default in a 3.x minor. Anything that parses today
would start throwing, so I do not think the 2.x default should change.

https://github.com/FasterXML/jackson-core/pull/1684#issuecomment-5612240477

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.