apache / apache/hudi

[BUG] TIMESTAMP partition column under slash partitioning: SqlKeyGenerator slash arm shadows the TimestampType normalization

Open
#19,667 1 comment 0 reactions 1 assignee Claimed by @SEPURI-SAI-KRISHNA View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Bug Description

**What happened:**

On a table with `hoodie.datasource.write.slash.separated.date.partitioning=true` and a
`TIMESTAMP` partition column, the Avro write path and the row-writer path derive different
partition values.

`SqlKeyGenerator`'s slash-partitioning arm sits ahead of the `TimestampType` normalization in the
same `dataType` match, so it shadows it. With the default
`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=false`, the Avro path
writes the raw micros value (for example `1767607200000000`) while the row-writer path writes the
normalized rendering (`2026/01/05 18:00:00.0`).

Two write paths on the same table therefore place records for the same logical value into
different directories.

**What you expected:**

Either both paths agree on the partition value for a TIMESTAMP column, or a TIMESTAMP partition
column is rejected under slash-separated date partitioning -- the feature is documented for
`yyyy-MM-dd` date values.

**Steps to reproduce:**
1. Create a table with a `TIMESTAMP` partition column and
`hoodie.datasource.write.slash.separated.date.partitioning=true`.
2. Write one batch through the Avro path and one through the row writer
(`hoodie.spark.sql.insert.into.operation=bulk_insert`).
3. Compare `_hoodie_partition_path` and the directories on disk between the two batches.

**Suggested fix:**

Move the slash arm inside the `case _ =>` branch of the `dataType` match in `SqlKeyGenerator`, so
the `TimestampType` normalization keeps precedence; or reject `TimestampType` partition columns
when slash-separated date partitioning is enabled.

Raised during review of #19648.

## Environment

**Hudi version:** master (1.3.0-SNAPSHOT)
**Query engine:** Spark
**Relevant configs:** `hoodie.datasource.write.slash.separated.date.partitioning=true`,
`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=false` (default),
TIMESTAMP partition column

## Logs and Stack Trace

No failure -- the two write paths just disagree on the directory.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.