Followup for fixing negative sentinel fields for VARIANT and BLOB in InternalSchemaConverter
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
In the following PR review discussed with @yihua the below
* https://github.com/apache/hudi/pull/18538#discussion_r3121944778
* https://github.com/apache/hudi/pull/18538#discussion_r3126089651
```
@yihua correct its just an id and any negative value can work. The negative constraint is what prevents collision with user-assigned IDs (which are all positive integers in the auto-assignment path).
-10/-11/-12 specifically is just to leave gap after Variant's -1/-2 so future changes to variant shape can claim potentially the -3, -4, -5.
```
Ideally we should avoid using this negative ids for fields value in future and not match in length of the array fields like this
```
// Detect Variant round-trip: sentinel negative IDs with value/metadata fields
if (fields.size() == 2) {
Types.Field field0 = fields.get(0);
Types.Field field1 = fields.get(1);
boolean hasNegativeIds = field0.fieldId() < 0 && field1.fieldId() < 0;
boolean hasVariantFields = (field0.name().equals(HoodieSchema.Variant.VARIANT_VALUE_FIELD)
&& field1.name().equals(HoodieSchema.Variant.VARIANT_METADATA_FIELD))
|| (field0.name().equals(HoodieSchema.Variant.VARIANT_METADATA_FIELD)
&& field1.name().equals(HoodieSchema.Variant.VARIANT_VALUE_FIELD));
if (hasNegativeIds && hasVariantFields) {
// TODO: Flesh out schema evolution for Variant types #18285
return HoodieSchema.createVariant();
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the InternalSchemaConverter logic shown in the issue and review the linked PR #18538 discussions, then trace HoodieSchema.createVariant() and the handling of VARIANT and BLOB fields. Clarify how sentinel field IDs should be recognized without relying on array length; done means the converter no longer depends on the current negative-ID and two-field assumptions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100