C and Pure parser accept invalid UTF-8 strings, the Java parser doesn't.
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 782
- Forks
- 383
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 10
Description
I know, I know...if it's bad content it's bad content. But this represents a difference from MRI.
Here's the case, again a reduced version of one I got from @rkh:
# encoding: utf-8
require 'json'
x = "{\"foo\":\"\xC3\"}"
h = JSON.parse(x)
p h['foo']
p h['foo'].encoding
So basically there's a bad byte in a UTF-8 string, and the MRI version walks right by it and allows it to come through to the resulting parsed json structure.
I have a totally broken patch for this:
diff --git a/java/src/json/ext/ByteListTranscoder.java b/java/src/json/ext/ByteListTranscoder.java
index ed9e54b..a7e42ba 100644
--- a/java/src/json/ext/ByteListTranscoder.java
+++ b/java/src/json/ext/ByteListTranscoder.java
@@ -78,9 +78,10 @@ abstract class ByteListTranscoder {
return head;
}
if (head <= 0xbf) { // 0b10xxxxxx
- throw invalidUtf8(); // tail byte with no head
+ return head; //throw invalidUtf8(); // tail byte with no head
}
if (head <= 0xdf) { // 0b110xxxxx
+ if (pos + 1 > srcEnd) return head;
ensureMin(1);
int cp = ((head & 0x1f) << 6)
| nextPart();
Again, I'm not sure this is actually something that needs to be fixed, but because the MRI version of json does not blow up on this content, there's something to be addressed.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the Ruby example first, then read java/src/json/ext/ByteListTranscoder.java and compare the C, Pure, and Java parser behavior for invalid UTF-8. Clarify which behavior should match MRI before changing the parser; done means the implementations have an agreed, consistent result for this input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, java, ruby
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100