apache / apache/hudi

[BUG] hive_style_partitioning + slash.separated.date.partitioning accepted outside SQL DDL despite the config documenting them as exclusive

Open
#19,669 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Bug Description

**What happened:**

The documentation of `hoodie.datasource.write.slash.separated.date.partitioning` states the
constraint outright:

> "Flag to indicate whether to use slash separated date partitioning. If set to true, date
> partition values in yyyy-MM-dd format will be transformed to yyyy/MM/dd directory structure.
> By default false. **Cannot be used together with hive-style partitioning.**"

Only SQL DDL enforces it. `HoodieCatalogTable#extraTableConfig` rejects the combination by
inspecting `sqlOptions`, but `HoodieWriterUtils`, `HoodieSparkSqlWriter` and `BaseKeyGenerator`
have no equivalent check, so `df.write.format("hudi")` and HoodieStreamer accept
`hive_style_partitioning=true` together with `slash.separated.date.partitioning=true`.

For a single partition field the two write paths then disagree, and both are live (Avro versus
Spark record type, plus the row writer):

| path | partition directory |
|---|---|
| Avro (`KeyGenUtils#getPartitionPath`) | `date_col=2026/01/05` |
| Row / InternalRow (`PartitionPathFormatterBase`) | `date_col=2026-01-05` |

The Avro layout is also not readable back:
`HoodieSparkUtils#doParsePartitionColumnValues` takes the `date_col=` prefix branch and never
restores `/` to `-`.

**What you expected:**

A documented-invalid combination should be rejected by every writer, not only by SQL DDL.

**Steps to reproduce:**
1. `df.write.format("hudi")` with both `hoodie.datasource.write.hive_style_partitioning=true` and
`hoodie.datasource.write.slash.separated.date.partitioning=true`, one date partition column.
2. The write is accepted; compare the directory produced by the Avro path against the row-writer
one, and read the Avro-written table back.

**Suggested fix:**

Add the mutual-exclusion check to `HoodieWriterUtils.validateTableConfig` so it applies to every
write path, matching what the config documentation already promises.

Raised during review of #19648.

## Environment

**Hudi version:** master (1.3.0-SNAPSHOT)
**Query engine:** Spark (`df.write` / HoodieStreamer, i.e. not SQL DDL)
**Relevant configs:** `hoodie.datasource.write.hive_style_partitioning=true` +
`hoodie.datasource.write.slash.separated.date.partitioning=true`

## Logs and Stack Trace

No failure at write time -- the layouts simply diverge, and the Avro-written one does not read
back correctly.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read HoodieWriterUtils.validateTableConfig and compare it with HoodieCatalogTable#extraTableConfig; trace the df.write.format("hudi") and HoodieStreamer entry points. Reproduce the write with both configs enabled, then verify every writer rejects the combination, including non-SQL paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.