SpaceCurveSortingHelper single-column path only range-partitions; rows are not sorted within partitions
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
`SpaceCurveSortingHelper.orderDataFrameByMappingValues` handles a single ordering column with a short-circuit:
```java
// In case when there's just one column to be ordered by, we can skip space-curve
// ordering altogether (since it will match linear ordering anyway)
if (orderByCols.size() == 1) {
...
// TODO validate if we need Spark to re-partition
return df.repartitionByRange(targetPartitionCount, new Column(orderByColName));
}
```
The comment claims the result "will match linear ordering anyway", but `repartitionByRange` only guarantees non-overlapping ranges across partitions. Per the Spark javadoc, "the rows are not sorted in each partition of the resulting Dataset."
The multi-column ZORDER/HILBERT paths in the same method do a real `RDD.sortBy`, so rows within each output partition (and therefore within each written file) are fully sorted by the curve ordinal. The single-column path leaves rows inside each partition in arbitrary order, so files produced by layout optimization on a single column are range-separated across files but unsorted within a file. That weakens page/row-group level min-max stats compared to the intended linear ordering.
This was surfaced while adding unit coverage in #19219: a test asserting ascending output for the single-column path failed deterministically in CI (rows came back in insertion order).
**To Reproduce**
Call `SpaceCurveSortingHelper.orderDataFrameByMappingValues(df, ZORDER, singletonList("c1"), 1)` on an unsorted frame and collect: the output preserves input order instead of ascending `c1` order.
**Expected behavior**
Either the single-column short-circuit should actually produce linear ordering (e.g. sort within partitions after the range repartition, matching the sort guarantee of the multi-column paths), or the comment/documentation should be corrected to state that only cross-partition range separation is provided.
**Environment Description**
* Hudi version : master
* Spark version : all supported
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.