apache / apache/hudi

BLOB column should be rejected as primaryKey / recordKey

Open
#18,819 3 comments 0 reactions 1 assignee Claimed by @ZZZxDong View on GitHub
type:bug type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**What happened:**
`CREATE TABLE ... TBLPROPERTIES (primaryKey = '')` followed by INSERT both succeed silently when the keyed column is of BLOB type. The resulting `_hoodie_record_key` is the JSON-stringified BLOB struct, e.g. `{"type":"INLINE","data":"hello-0","reference":null}`.

BLOB is raw binary bytes (images, video, embeddings, or EXTERNAL references to such payloads). It is not a valid record-key type semantically:
- For INLINE BLOBs, the key is the entire byte payload — for real-world blobs (MB-sized images/video/embeddings) the key balloons proportionally, blowing up shuffle bytes and metadata index (record index, secondary index, bloom) storage.
- For EXTERNAL BLOBs, the key is derived from the storage path, so record identity tracks path rather than content — moving or re-uploading the same blob yields a different key.

**What you expected:**
Hudi should reject BLOB-typed columns as the record key, the same way other unsupported key types are rejected.

- Spark DDL: `CREATE TABLE ... TBLPROPERTIES (primaryKey = '')`
- Spark DataSource writes: `.option("hoodie.datasource.write.recordkey.field", "")`

Both should fail fast with a clear error message identifying the BLOB column and the unsupported-type reason.

**Steps to reproduce:**
1. Use 1.2.0 Spark bundle.
2. Either:
a. DDL path: `CREATE TABLE t (id BLOB, label STRING) USING hudi TBLPROPERTIES (primaryKey = 'id')`
b. DataSource path: `df.write.format("hudi").option("hoodie.datasource.write.recordkey.field", "id").save(...)` with `id` of BLOB type.
3. INSERT / write a row with an INLINE BLOB value.
4. `SELECT _hoodie_record_key FROM t` → key is the JSON-serialized struct.

**Environment:**
- Hudi version: 1.2.0-rc2
- Query engine: Spark 3.5

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.