apache / apache/hudi

Followup for fixing negative sentinel fields for VARIANT and BLOB in InternalSchemaConverter

Open
#18,551 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

In the following PR review discussed with @yihua the below
* https://github.com/apache/hudi/pull/18538#discussion_r3121944778
* https://github.com/apache/hudi/pull/18538#discussion_r3126089651

```
@yihua correct its just an id and any negative value can work. The negative constraint is what prevents collision with user-assigned IDs (which are all positive integers in the auto-assignment path).

-10/-11/-12 specifically is just to leave gap after Variant's -1/-2 so future changes to variant shape can claim potentially the -3, -4, -5.

```

Ideally we should avoid using this negative ids for fields value in future and not match in length of the array fields like this

```
// Detect Variant round-trip: sentinel negative IDs with value/metadata fields
if (fields.size() == 2) {
Types.Field field0 = fields.get(0);
Types.Field field1 = fields.get(1);
boolean hasNegativeIds = field0.fieldId() < 0 && field1.fieldId() < 0;
boolean hasVariantFields = (field0.name().equals(HoodieSchema.Variant.VARIANT_VALUE_FIELD)
&& field1.name().equals(HoodieSchema.Variant.VARIANT_METADATA_FIELD))
|| (field0.name().equals(HoodieSchema.Variant.VARIANT_METADATA_FIELD)
&& field1.name().equals(HoodieSchema.Variant.VARIANT_VALUE_FIELD));

if (hasNegativeIds && hasVariantFields) {
// TODO: Flesh out schema evolution for Variant types #18285
return HoodieSchema.createVariant();
}
}

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the InternalSchemaConverter logic shown in the issue and review the linked PR #18538 discussions, then trace HoodieSchema.createVariant() and the handling of VARIANT and BLOB fields. Clarify how sentinel field IDs should be recognized without relying on array length; done means the converter no longer depends on the current negative-ID and two-field assumptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.