protocolbuffers / protocolbuffers/protobuf
upb parses float/double field defaults with locale-dependent strtod, breaking descriptor loading under a comma-decimal LC_NUMERIC
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 72k
- Forks
- 16.3k
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 140
Description
What version of protobuf and what language are you using?
Version: v7.35.1 (Python, upb backend)
Language: Python
What operating system (Linux, Windows, ...) and version?
Linux (glibc), reproduced with LC_NUMERIC=fr_FR.UTF-8
What runtime / compiler are you using (e.g., python version or gcc version)
CPython 3.12
What did you do?
Load any .proto-generated descriptor that has a float/double field with a
non-integral default value, while the process' LC_NUMERIC is a locale that uses
, as the decimal separator.
import locale
locale.setlocale(locale.LC_NUMERIC, "fr_FR.UTF-8")
# TrainerSpec.character_coverage has [default = 0.9995]
import sentencepiece.sentencepiece_model_pb2
What did you expect to see?
The descriptor loads. Default values in a FileDescriptorProto are part of the
schema and are always written with a . decimal separator, so parsing them
should not depend on the process locale.
What did you see instead?
TypeError: Couldn't build proto file into descriptor pool:
Invalid default '0.9995' for field sentencepiece.TrainerSpec.character_coverage of type 2
With LC_NUMERIC=C the same code succeeds and yields 0.9994999766349792.
Analysis
upb/reflection/field_def.c parses the default value with the locale-sensitive
C library functions:
- https://github.com/protocolbuffers/protobuf/blob/main/upb/reflection/field_def.c#L486 —
double val = strtod(str, &end); - https://github.com/protocolbuffers/protobuf/blob/main/upb/reflection/field_def.c#L494 —
float val = strtof(str, &end);
Under a comma-decimal locale, strtod("0.9995", &end) stops at the ., leaving
*end != '\0', so the parse is rejected and the invalid: branch reports
Invalid default.
The C++ full runtime avoids this deliberately by routing the same parse through a
locale-independent helper (io::NoLocaleStrtod). upb used to carry an equivalent
_upb_NoLocaleStrtod() in upb/lex/strtod.c, but it was removed in #26377 on the
grounds that it was unused — which is accurate, but the underlying need is still
there in the descriptor builder.
Suggested fix: parse the default with a locale-independent conversion (or
temporarily normalize the radix character) in parse_default(), so that upb
matches the C++ runtime's behavior.
Impact
This is not limited to a single library, but it is easiest to hit through
sentencepiece, whose sentencepiece_model.proto has several float defaults.
Any Python process that sets a comma-decimal LC_NUMERIC before importing a
sentencepiece-based tokenizer fails to load it. Notably, Qt applications hit this
without doing anything themselves: QApplication's constructor calls
setlocale(LC_ALL, "") on Unix, so every Qt-based Python GUI on a machine with a
European locale cannot load any sentencepiece tokenizer.
Anything else we should know about your project / environment?
Workaround for affected applications is locale.setlocale(locale.LC_NUMERIC, "C")
before loading descriptors.
Investigation and draft by Claude (Claude Code); repro run and reviewed by me.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in upb/reflection/field_def.c at parse_default(), especially the strtod and strtof calls identified in the issue. Reproduce the descriptor-loading failure with LC_NUMERIC=fr_FR.UTF-8 and compare it with LC_NUMERIC=C. Done means float and double defaults load consistently under comma-decimal locales while retaining the expected parsed values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100