BLOB column should be rejected as primaryKey / recordKey
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**What happened:**
`CREATE TABLE ... TBLPROPERTIES (primaryKey = '')` followed by INSERT both succeed silently when the keyed column is of BLOB type. The resulting `_hoodie_record_key` is the JSON-stringified BLOB struct, e.g. `{"type":"INLINE","data":"hello-0","reference":null}`.
BLOB is raw binary bytes (images, video, embeddings, or EXTERNAL references to such payloads). It is not a valid record-key type semantically:
- For INLINE BLOBs, the key is the entire byte payload — for real-world blobs (MB-sized images/video/embeddings) the key balloons proportionally, blowing up shuffle bytes and metadata index (record index, secondary index, bloom) storage.
- For EXTERNAL BLOBs, the key is derived from the storage path, so record identity tracks path rather than content — moving or re-uploading the same blob yields a different key.
**What you expected:**
Hudi should reject BLOB-typed columns as the record key, the same way other unsupported key types are rejected.
- Spark DDL: `CREATE TABLE ... TBLPROPERTIES (primaryKey = '')`
- Spark DataSource writes: `.option("hoodie.datasource.write.recordkey.field", "")`
Both should fail fast with a clear error message identifying the BLOB column and the unsupported-type reason.
**Steps to reproduce:**
1. Use 1.2.0 Spark bundle.
2. Either:
a. DDL path: `CREATE TABLE t (id BLOB, label STRING) USING hudi TBLPROPERTIES (primaryKey = 'id')`
b. DataSource path: `df.write.format("hudi").option("hoodie.datasource.write.recordkey.field", "id").save(...)` with `id` of BLOB type.
3. INSERT / write a row with an INLINE BLOB value.
4. `SELECT _hoodie_record_key FROM t` → key is the JSON-serialized struct.
**Environment:**
- Hudi version: 1.2.0-rc2
- Query engine: Spark 3.5
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.