[BUG] TIMESTAMP partition column under slash partitioning: SqlKeyGenerator slash arm shadows the TimestampType normalization
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
## Bug Description
**What happened:**
On a table with `hoodie.datasource.write.slash.separated.date.partitioning=true` and a
`TIMESTAMP` partition column, the Avro write path and the row-writer path derive different
partition values.
`SqlKeyGenerator`'s slash-partitioning arm sits ahead of the `TimestampType` normalization in the
same `dataType` match, so it shadows it. With the default
`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=false`, the Avro path
writes the raw micros value (for example `1767607200000000`) while the row-writer path writes the
normalized rendering (`2026/01/05 18:00:00.0`).
Two write paths on the same table therefore place records for the same logical value into
different directories.
**What you expected:**
Either both paths agree on the partition value for a TIMESTAMP column, or a TIMESTAMP partition
column is rejected under slash-separated date partitioning -- the feature is documented for
`yyyy-MM-dd` date values.
**Steps to reproduce:**
1. Create a table with a `TIMESTAMP` partition column and
`hoodie.datasource.write.slash.separated.date.partitioning=true`.
2. Write one batch through the Avro path and one through the row writer
(`hoodie.spark.sql.insert.into.operation=bulk_insert`).
3. Compare `_hoodie_partition_path` and the directories on disk between the two batches.
**Suggested fix:**
Move the slash arm inside the `case _ =>` branch of the `dataType` match in `SqlKeyGenerator`, so
the `TimestampType` normalization keeps precedence; or reject `TimestampType` partition columns
when slash-separated date partitioning is enabled.
Raised during review of #19648.
## Environment
**Hudi version:** master (1.3.0-SNAPSHOT)
**Query engine:** Spark
**Relevant configs:** `hoodie.datasource.write.slash.separated.date.partitioning=true`,
`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=false` (default),
TIMESTAMP partition column
## Logs and Stack Trace
No failure -- the two write paths just disagree on the directory.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.