Improve JSON text to BinaryJSON conversion performance
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
`ParseBinaryJSONFromString` currently validates and converts JSON text in three stages:
1. scan the complete input with `json.Valid`;
2. decode it into a temporary `any` tree with `encoding/json.Decoder`;
3. walk that tree again in `CreateBinaryJSONWithCheck` to produce `BinaryJSON`.
Profiles of JSON import workloads show that this repeated scanning, string unescaping, temporary tree construction, and final conversion can consume significant CPU and allocations.
Parse valid JSON text directly into compact node metadata and encode the final `BinaryJSON` into one exactly sized buffer. The optimized path should preserve the existing behavior for:
- all JSON scalar and container types;
- integer, unsigned integer, and floating-point boundaries;
- duplicate object keys (the last value wins) and binary key ordering;
- escaped strings, invalid UTF-8 replacement, maximum key length, and maximum document depth;
- `BinaryJSON.UnmarshalJSON` accepting bytes after the first JSON value, as its current `Decoder.Decode` implementation does;
- malformed-input error behavior.
A local prototype using jsoniter only as the JSON tokenizer shows the following results on Apple M3 (`go test`, 300 ms benchtime). The allocation counts are deterministic; timings are representative medians from repeated runs.
| Input | Existing | Prototype | Existing allocs | Prototype allocs |
| --- | ---: | ---: | ---: | ---: |
| Small mixed document | 1.72 us | 0.79 us | 31 | 10 |
| All JSON types | 3.28 us | 1.72 us | 50 | 26 |
| Object with 32 numbers | 11.53 us | 4.95 us | 122 | 70 |
| Object with 32 strings | 19.22 us | 4.04 us | 124 | 68 |
| 32 nested objects | 41.54 us | 18.48 us | 547 | 232 |
Contributor guide
Research direction
Start with ParseBinaryJSONFromString, json.Valid, encoding/json.Decoder, and CreateBinaryJSONWithCheck to map the current three-stage conversion. Review BinaryJSON.UnmarshalJSON for its post-value behavior and the listed boundary, duplicate-key, depth, UTF-8, and malformed-input requirements. Compare the optimized path against the existing behavior and prototype benchmarks when all cases remain compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100