apache / apache/hudi

Check custom type edge case

Open
#18,603 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

@voonhous LGTM but can you check this one weird case in case a user would try this (unlikley but sharing below):

```

The validator code

(descriptorType, f.dataType) match {
case (HoodieSchemaType.BLOB, st: StructType) => validateBlobStructure(st)
case (HoodieSchemaType.VARIANT, st: StructType) => validateVariantStructure(st)
case _ => // <-- silently no-op
}

The pattern only matches when both the tag says BLOB and the data type is a
StructType. Anything else falls into case _ and does nothing.

The bug, concretely

Suppose a user (or a buggy upstream transform) builds this schema:

val blobMetadata = new MetadataBuilder()
.putString(HoodieSchema.TYPE_METADATA_FIELD, HoodieSchemaType.BLOB.name())
.build()

val schema = new StructType()
.add("id", LongType)
.add("payload", LongType, nullable = true, metadata = blobMetadata)
// ^^^^^^^^ ^^^^^^^^^^^^
// wrong type says "I'm a BLOB"

The user is asserting "payload is a BLOB" via the metadata, but the data type is a
LongType, not the canonical BLOB struct.

What happens today

1. validateCustomTypeStructures(schema) runs.
2. It sees the hudi_type=BLOB tag on payload.
3. The match tuple is (BLOB, LongType) — neither pattern matches → falls into case _
→ returns without throwing.
4. Then convertStructTypeToHoodieSchema runs.
5. The BLOB case in toHoodieTypeNested is case blobStruct: StructType if
metadata.contains(...) && ...isCanonicalBlobStruct(blobStruct) => — requires a
StructType, so it doesn't match either.
6. LongType falls through to the normal case LongType => HoodieSchema.create(LONG)
arm.
7. Result: the field is silently written as a plain LONG. The BLOB tag is ignored,
no error.

The user thinks they wrote a BLOB column; the table actually has a LONG column.

The fix

Add an explicit reject for "tag says BLOB/VARIANT but the type is wrong":

(descriptorType, f.dataType) match {
case (HoodieSchemaType.BLOB, st: StructType) => validateBlobStructure(st)
case (HoodieSchemaType.VARIANT, st: StructType) => validateVariantStructure(st)
case (HoodieSchemaType.BLOB, other) =>
throw new IllegalArgumentException(
s"Field '${f.name}' is tagged hudi_type=BLOB but has type $other; expected a
StructType.")
case (HoodieSchemaType.VARIANT, other) =>
throw new IllegalArgumentException(
s"Field '${f.name}' is tagged hudi_type=VARIANT but has type $other; expected
a StructType.")
case _ =>
}

Now the misuse fails fast at the write boundary instead of silently producing the
wrong on-disk schema.
```

_Originally posted by @rahil-c in https://github.com/apache/hudi/issues/18566#issuecomment-4322589008_

Contributor guide

No contributing guide indexed for this repository

Research direction

Search for validateCustomTypeStructures, validateBlobStructure, validateVariantStructure, and toHoodieTypeNested to locate the validator and conversion path. Reproduce the tagged LongType example, then run the relevant schema tests if present; done means mismatched BLOB or VARIANT metadata fails instead of being converted as a plain type.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.