apache / apache/hudi

[BUG] slash.separated.date.partitioning with more than one partition field breaks reads (empty results, or HoodieException with CustomKeyGenerator)

Open
#19,666 2 comments 0 reactions 1 assignee Claimed by @SEPURI-SAI-KRISHNA View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Bug Description

**What happened:**

`hoodie.datasource.write.slash.separated.date.partitioning=true` is accepted on a table with
more than one partition field, but the write and the query-side path composition disagree. The
failure mode differs by key generator, and both are bad.

### `SimpleKeyGenerator` / `ComplexKeyGenerator`: silent zero rows

For a `(datestr, city)` table written by the Avro path, `KeyGenUtils#getRecordPartitionPath`
guards the substitution on `partitionPathFields.size() == 1`, so the directory is
`2026-01-05/san-francisco`.

On the query side, `SparkHoodieTableFileIndex#composeRelativePartitionPath` builds a
`StringPartitionPathFormatter` from table config and combines only the bound *prefix* of the
partition columns. A query filtering on `datestr` alone composes the single-part value, the
formatter's single-field branch applies the dash-to-slash substitution, and the resulting
`2026/01/05` does not exist on disk. `exists()` fails and the query returns nothing.

### `CustomKeyGenerator`: every read throws

`CustomKeyGenerator` builds one single-field sub-key-generator per partition field, so the
`size() == 1` guard above never fires and *every* field gets slash-separated. A
`datestr:simple,city:simple` table writes `2026/01/05/san/francisco` — consistently, on the
Avro, `Row` and `InternalRow` paths alike, so this is not a write-path disagreement.

The layout simply cannot be read back. `HoodieSparkUtils#doParsePartitionColumnValues` sees 5
path fragments for 2 partition columns with no `col=` prefix to key off, and returns a
length-0 array. Under the default lazy listing that trips the length check in
`BaseHoodieTableFileIndex#getPartitionColumnValues`:

```java
if (shouldListLazily && partitionColumnValues.length != partitionColumns.length) {
throw new HoodieException("Failed to parse partition column values from the partition-path:"
+ " likely non-encoded slashes being used in partition column's values. ...");
}
```

so reading the table throws `HoodieException` rather than returning wrong rows.

### The invariant is already asserted elsewhere

`ShowHoodieTablePartitionsCommand.scala:60-61` states the requirement this configuration
violates:

```scala
ValidationUtils.checkState(partitionColumnNamesOpt.get().length == 1,
"Only one partition field is allowed for SlashEncodedPartitioning")
```

so one code path treats single-field as a hard requirement while table creation accepts any
number of fields.

**What you expected:**

Either the combination is rejected at table creation and in writer config validation, or all
paths agree on a layout that can be read back.

**Steps to reproduce:**

`SimpleKeyGenerator` (silent empty result):
1. Create a COW table partitioned by `(datestr, city)` with
`hoodie.datasource.write.slash.separated.date.partitioning=true`.
2. Insert a row with `datestr='2026-01-05'`, `city='san-francisco'`.
3. `SELECT * FROM t WHERE datestr = '2026-01-05'` -- returns zero rows, while a query binding
both partition columns returns the row.

`CustomKeyGenerator` (hard failure):
1. Same table, with `hoodie.datasource.write.keygenerator.class` set to `CustomKeyGenerator`
and `hoodie.datasource.write.partitionpath.field=datestr:simple,city:simple`.
2. Insert the same row -- the directory written is `2026/01/05/san/francisco`.
3. `SELECT * FROM t` -- throws `HoodieException` ("Failed to parse partition column values
from the partition-path") under the default lazy listing.

**Suggested fix:**

Reject `slash.separated.date.partitioning=true` together with more than one partition field, next
to the existing hive-style check in `HoodieCatalogTable#extraTableConfig` and in
`HoodieWriterUtils.validateTableConfig` so non-SQL writers are covered too. A check on the
partition-field count covers `CustomKeyGenerator` as well, since the count is a table-level
property and does not depend on how the key generator decomposes it into sub-key-generators.

## Environment

**Hudi version:** master (1.3.0-SNAPSHOT)
**Query engine:** Spark
**Relevant configs:** `hoodie.datasource.write.slash.separated.date.partitioning=true` with two or
more partition fields; `CustomKeyGenerator` with more than one `field:type` pair

## Logs and Stack Trace

`SimpleKeyGenerator`/`ComplexKeyGenerator`: no failure -- the query silently returns an empty result.

`CustomKeyGenerator`: `org.apache.hudi.exception.HoodieException: Failed to parse partition column
values from the partition-path: likely non-encoded slashes being used in partition column's values.
You can try to work this around by switching listing mode to eager`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.