Protobuf transform handling with default values
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
Druid 33.0.0
### Description
There appears to be an issue in the Protobuf transform handling where default-valued fields in non-optional proto definitions are being set to null during ingestion.
Observed behavior:
- When a non-optional proto field (for example, an int32 with a default value of 0) is used inside a transform expression, the resulting value becomes null instead of retaining its default value.
- This behavior does **not** occur with optional fields. Optional fields behave as expected.
- Because of this, we are forced to wrap non-optional proto fields with functions like `nvl()` or similar null-handling expressions to avoid unexpected nulls in the ingestion pipeline.
From code inspection:
- I think this problem should be in convertMessage on /extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/ProtobufConverter.java
- It looks like the getAllFields() logic, when a value is set to default, treats default proto values (like 0 for int32) as equivalent to "not set", resulting in the field being mapped to null.
- This appears inconsistent with Protobuf semantics, where default numeric values are valid values and not indicators of missing fields.
Contributor guide
Research direction
Start in extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/ProtobufConverter.java, focusing on convertMessage and its getAllFields() handling. Reproduce the difference between non-optional default-valued fields and optional fields, then verify that non-optional defaults are retained during transform ingestion while optional-field behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100