protocolbuffers / protocolbuffers/protobuf

upb parses float/double field defaults with locale-dependent strtod, breaking descriptor loading under a comma-decimal LC_NUMERIC

Open
#28,796 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
72k
Forks
16.3k
Avg merge
1d 17h
Merged PRs (30d)
140

Description

What version of protobuf and what language are you using?

Version: v7.35.1 (Python, upb backend)
Language: Python

What operating system (Linux, Windows, ...) and version?

Linux (glibc), reproduced with LC_NUMERIC=fr_FR.UTF-8

What runtime / compiler are you using (e.g., python version or gcc version)

CPython 3.12

What did you do?

Load any .proto-generated descriptor that has a float/double field with a
non-integral default value, while the process' LC_NUMERIC is a locale that uses
, as the decimal separator.

import locale
locale.setlocale(locale.LC_NUMERIC, "fr_FR.UTF-8")

# TrainerSpec.character_coverage has [default = 0.9995]
import sentencepiece.sentencepiece_model_pb2
What did you expect to see?

The descriptor loads. Default values in a FileDescriptorProto are part of the
schema and are always written with a . decimal separator, so parsing them
should not depend on the process locale.

What did you see instead?
TypeError: Couldn't build proto file into descriptor pool:
Invalid default '0.9995' for field sentencepiece.TrainerSpec.character_coverage of type 2

With LC_NUMERIC=C the same code succeeds and yields 0.9994999766349792.

Analysis

upb/reflection/field_def.c parses the default value with the locale-sensitive
C library functions:

Under a comma-decimal locale, strtod("0.9995", &end) stops at the ., leaving
*end != '\0', so the parse is rejected and the invalid: branch reports
Invalid default.

The C++ full runtime avoids this deliberately by routing the same parse through a
locale-independent helper (io::NoLocaleStrtod). upb used to carry an equivalent
_upb_NoLocaleStrtod() in upb/lex/strtod.c, but it was removed in #26377 on the
grounds that it was unused — which is accurate, but the underlying need is still
there in the descriptor builder.

Suggested fix: parse the default with a locale-independent conversion (or
temporarily normalize the radix character) in parse_default(), so that upb
matches the C++ runtime's behavior.

Impact

This is not limited to a single library, but it is easiest to hit through
sentencepiece, whose sentencepiece_model.proto has several float defaults.
Any Python process that sets a comma-decimal LC_NUMERIC before importing a
sentencepiece-based tokenizer fails to load it. Notably, Qt applications hit this
without doing anything themselves: QApplication's constructor calls
setlocale(LC_ALL, "") on Unix, so every Qt-based Python GUI on a machine with a
European locale cannot load any sentencepiece tokenizer.

Anything else we should know about your project / environment?

Workaround for affected applications is locale.setlocale(locale.LC_NUMERIC, "C")
before loading descriptors.


Investigation and draft by Claude (Claude Code); repro run and reviewed by me.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in upb/reflection/field_def.c at parse_default(), especially the strtod and strtof calls identified in the issue. Reproduce the descriptor-loading failure with LC_NUMERIC=fr_FR.UTF-8 and compare it with LC_NUMERIC=C. Done means float and double defaults load consistently under comma-decimal locales while retaining the expected parsed values.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.