apache / apache/druid

Protobuf transform handling with default values

Open
#18,792 1 comment 0 reactions 0 assignees View on GitHub
stale Uncategorized problem report
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

Druid 33.0.0

### Description

There appears to be an issue in the Protobuf transform handling where default-valued fields in non-optional proto definitions are being set to null during ingestion.

Observed behavior:
- When a non-optional proto field (for example, an int32 with a default value of 0) is used inside a transform expression, the resulting value becomes null instead of retaining its default value.
- This behavior does **not** occur with optional fields. Optional fields behave as expected.
- Because of this, we are forced to wrap non-optional proto fields with functions like `nvl()` or similar null-handling expressions to avoid unexpected nulls in the ingestion pipeline.

From code inspection:
- I think this problem should be in convertMessage on /extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/ProtobufConverter.java
- It looks like the getAllFields() logic, when a value is set to default, treats default proto values (like 0 for int32) as equivalent to "not set", resulting in the field being mapped to null.
- This appears inconsistent with Protobuf semantics, where default numeric values are valid values and not indicators of missing fields.

Contributor guide

Open the contributing guide

Research direction

Start in extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/ProtobufConverter.java, focusing on convertMessage and its getAllFields() handling. Reproduce the difference between non-optional default-valued fields and optional fields, then verify that non-optional defaults are retained during transform ingestion while optional-field behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.