apache / apache/hudi

[BUG] InternalSchema BLOB reference field ids collide with table field ids

Open
#19,833 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

`InternalSchemaConverter.buildBlobInternalRecordType()` (`InternalSchemaConverter.java:70-87`, since #18538) gives the BLOB sub-fields sentinel ids `-10/-11/-12`, but the nested `reference` record's four fields get ids `0`, `1`, `2`, `3`. `InternalSchema.buildIdToField` is a single flat map over the whole tree, filled child-first with last-put-wins (`InternalSchemaBuilder.java:150-156`), so every table that has a BLOB column also has `reference.external_path` registered under id `0`, `reference.offset` under `1`, and so on -- the same ids the table's own leading columns carry.

Any id-based InternalSchema operation on such a table resolves the wrong field. Through `HoodieSchemaUtils.asNullable` (`HoodieSchemaUtils.java:284-288`), which marks the still-required top-level columns nullable with a `ColumnUpdateChange`:

```java
HoodieSchema schema = HoodieSchema.createRecord("r", null, null, false, Arrays.asList(
HoodieSchemaField.of("id", HoodieSchema.create(HoodieSchemaType.INT), null, null),
HoodieSchemaField.of("b", HoodieSchema.createNullable(HoodieSchema.createBlob()), null, null)));
HoodieSchemaUtils.asNullable(schema).toAvroSchema().toString();
// {"type":"record","name":"r","fields":[{"name":"external_path","type":["null","string"],"default":null},{"name":"b",...
```

The `id` column comes back renamed to `external_path`: when the changed schema is rebuilt, field id 0 is looked up in the flat id map and resolves to the blob's nested `reference.external_path`, which was put last. With a required BLOB column instead (`HoodieSchema.createBlob()` without the nullable wrapper) the same call throws:

```
org.apache.hudi.exception.SchemaCompatibilityException: Cannot update nullability for column 'b' because it does not exist in the schema
at org.apache.hudi.common.schema.internal.action.TableChanges$ColumnUpdateChange.updateColumnNullability(TableChanges.java:192)
at org.apache.hudi.common.schema.HoodieSchemaUtils.lambda$asNullable$3(HoodieSchemaUtils.java:287)
```

Reachable from Flink clustering: `HoodieSchemaConverter.convertToSchema` emits `HoodieSchema.createBlob()` for a BLOB-shaped `RowType` (`hudi-flink-client/.../HoodieSchemaConverter.java:221-224`) and `ClusteringOperator.open()` passes that schema to `asNullable` (`ClusteringOperator.java:170-177`), so a Flink table with a BLOB column and a NOT NULL key gets a reader schema whose key column is renamed (nullable BLOB) or fails to open (required BLOB). The other InternalSchema entry points that see table schemas (`InternalSchemaCache`, `FileGroupReaderSchemaHandler`, `HoodieMergeHelper`, `BaseHoodieWriteClient` schema-on-read paths) share the same id map and are exposed to the same collision whenever a BLOB column is present.

Present since #18538 (`4ef56e4ebd79`). Not a regression from #19810, which keeps the conversion unchanged (the old `AvroSchemaUtils#asNullable` path produced the identical output; verified by running both against the schemas above).

Fix direction: allocate the `reference` field ids from the sentinel range as well (for example `-13..-16`) so no BLOB-internal id can collide with a table id, and make `buildIdToField` refuse duplicate ids (or assert on them) so the next fixed-shape type cannot reintroduce this. Tables that already persisted an InternalSchema with the colliding ids (schema-on-read enabled, BLOB column) need the reader to tolerate both id sets; that is the part that makes this a separate change rather than a one-line constant edit. Tests: `TestInternalSchemaConverter` round trip of a record with a leading required column plus a BLOB column asserting the leading column's name survives, and `TestHoodieSchemaUtils#asNullable` on the same shape.

### Environment

- Hudi master (`93f1f711e065`); Flink clustering on a table with a BLOB column is the reachable path

### Logs and Stack Trace

See above.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with InternalSchemaConverter.java:70-87 and InternalSchemaBuilder.java:150-156, then trace HoodieSchemaUtils.asNullable and the Flink ClusteringOperator path. Run TestInternalSchemaConverter and TestHoodieSchemaUtils#asNullable using a required leading field plus a BLOB field. Done means distinct internal ids, duplicate-id detection, correct handling of persisted colliding ids, and no renamed or missing columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.