pingcap / pingcap/tidb

Improve JSON text to BinaryJSON conversion performance

Open
#70,387 0 comments 0 reactions 0 assignees View on GitHub
component/expression type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

`ParseBinaryJSONFromString` currently validates and converts JSON text in three stages:

1. scan the complete input with `json.Valid`;
2. decode it into a temporary `any` tree with `encoding/json.Decoder`;
3. walk that tree again in `CreateBinaryJSONWithCheck` to produce `BinaryJSON`.

Profiles of JSON import workloads show that this repeated scanning, string unescaping, temporary tree construction, and final conversion can consume significant CPU and allocations.

Parse valid JSON text directly into compact node metadata and encode the final `BinaryJSON` into one exactly sized buffer. The optimized path should preserve the existing behavior for:

- all JSON scalar and container types;
- integer, unsigned integer, and floating-point boundaries;
- duplicate object keys (the last value wins) and binary key ordering;
- escaped strings, invalid UTF-8 replacement, maximum key length, and maximum document depth;
- `BinaryJSON.UnmarshalJSON` accepting bytes after the first JSON value, as its current `Decoder.Decode` implementation does;
- malformed-input error behavior.

A local prototype using jsoniter only as the JSON tokenizer shows the following results on Apple M3 (`go test`, 300 ms benchtime). The allocation counts are deterministic; timings are representative medians from repeated runs.

| Input | Existing | Prototype | Existing allocs | Prototype allocs |
| --- | ---: | ---: | ---: | ---: |
| Small mixed document | 1.72 us | 0.79 us | 31 | 10 |
| All JSON types | 3.28 us | 1.72 us | 50 | 26 |
| Object with 32 numbers | 11.53 us | 4.95 us | 122 | 70 |
| Object with 32 strings | 19.22 us | 4.04 us | 124 | 68 |
| 32 nested objects | 41.54 us | 18.48 us | 547 | 232 |

Contributor guide

Open the contributing guide

Research direction

Start with ParseBinaryJSONFromString, json.Valid, encoding/json.Decoder, and CreateBinaryJSONWithCheck to map the current three-stage conversion. Review BinaryJSON.UnmarshalJSON for its post-value behavior and the listed boundary, duplicate-key, depth, UTF-8, and malformed-input requirements. Compare the optimized path against the existing behavior and prototype benchmarks when all cases remain compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.