JSON: non-ASCII decimal digits in quoted numeric fields accepted by Python and Java, rejected by C++

The proto3 JSON mapping encodes 64-bit integers, and optionally other numeric types, as JSON strings. When that string contains non-ASCII decimal digits, the three first-party implementations do not agree: Python and Java accept them, C++ rejects them.

The same JSON document is therefore a valid message on two implementations and a parse error on the third.

Repro

probe.proto:

syntax = "proto3";
package probe;
message P {
  string s   = 1;
  int32  i32 = 3;
  int64  i64 = 4;
  uint32 u32 = 5;
  double d   = 6;
}

Python (protobuf runtime 7.36.2):

from google.protobuf import json_format
import probe_pb2 as M

json_format.Parse('{"i64":"123"}', M.P()).i64    # 123   <- ASCII control
json_format.Parse('{"i64":"١٢٣"}', M.P()).i64    # 123   <- Arabic-Indic U+0661-0663
json_format.Parse('{"i64":"1٢2"}', M.P()).i64    # 122   <- mixed ASCII + Arabic-Indic
json_format.Parse('{"i64":"१२३"}', M.P()).i64    # 123   <- Devanagari
json_format.Parse('{"i64":"abc"}', M.P())        # ParseError   <- negative control

C++ (libprotobuf 7.35.0, JsonStringToMessage), same inputs:

{"i64":"123"}   ACCEPTED  i64=123                      <- control
{"i64":"١٢٣"}   REJECTED  INVALID_ARGUMENT ... invalid number: '١٢٣'
{"i64":"1٢2"}   REJECTED  INVALID_ARGUMENT ... invalid number: '1٢2'
{"i64":"१२३"}   REJECTED  INVALID_ARGUMENT ... invalid number: '१२३'
{"i64":"abc"}   REJECTED                               <- negative control
{"s":"١٢٣"}     ACCEPTED  round-trips exactly          <- string field control

The string-field control matters: the same digits round-trip perfectly in a string field on every implementation, so this is specifically about the quoted-number path, not about UTF-8 handling generally.

Java (4.35.0) behaves like Python; that leg I measured through BigDecimal rather than reading the source.

The same divergence appears for int32, uint32 and double, and for Eastern Arabic-Indic (U+06F0-06F9), Thai, N'Ko and fullwidth digits. In C++ the double path reports a different error message at a different offset from the integer path, so there appear to be two independently ASCII-gated code paths rather than one.

Why this is reported as a divergence rather than a bug

The JSON mapping documentation says the value is a "decimal string" but does not define which characters are decimal. So I do not think either side is demonstrably violating the spec, and I am not arguing that C++ is wrong — arguably it is the safer behaviour. The problem is that the three implementations disagree, which makes the mapping non-portable for these inputs.

The lenient side is a silent data path rather than an error: {"i64":"1٢2"} becoming 122 on Python and Java is not obviously wrong to anyone reading the output. Arabic-Indic digits reach real systems through OCR, PDF extraction and copy-paste from localized UIs, so this is reachable input rather than a purely synthetic case.

Underlying cause on the lenient side is that the conversion delegates to a language primitive that is Unicode-aware by default — Python's int() accepts any character with Unicode_Decimal, and BigDecimal behaves similarly — whereas the C++ lexer gates on absl::ascii_isdigit.

What would resolve it

Either outcome seems fine as long as it is the same everywhere:

1. Define "decimal string" in the JSON mapping as ASCII 0-9 only, and tighten Python and Java to match C++; or
2. Define it as any Unicode decimal digit, and relax C++.

A conformance test covering a non-ASCII digit in a quoted numeric field would pin whichever is chosen.

Note

I could not find a prior report for this — searched issues and PRs, open and closed, including reading all text_format in:title issues. Closest related item I found is #29829, which is a similar accepted-leniency divergence between pure-Python and Java on base64 input.

Measured on macOS arm64. Python and C++ legs I ran myself; the Java leg is a single measurement and would be worth confirming independently.

Start by confirming the Java measurement, then trace quoted numeric parsing through C++ JsonStringToMessage and Python json_format; BigDecimal is the Java comparison point. Decide whether decimal strings permit only ASCII digits or Unicode decimal digits, then add a cross-implementation conformance test covering the reported numeric fields and mixed digits.