apache / apache/hudi

SpaceCurveSortingHelper single-column path only range-partitions; rows are not sorted within partitions

Open
#19,327 0 comments 0 reactions 1 assignee Claimed by @voonhous View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

`SpaceCurveSortingHelper.orderDataFrameByMappingValues` handles a single ordering column with a short-circuit:

```java
// In case when there's just one column to be ordered by, we can skip space-curve
// ordering altogether (since it will match linear ordering anyway)
if (orderByCols.size() == 1) {
...
// TODO validate if we need Spark to re-partition
return df.repartitionByRange(targetPartitionCount, new Column(orderByColName));
}
```

The comment claims the result "will match linear ordering anyway", but `repartitionByRange` only guarantees non-overlapping ranges across partitions. Per the Spark javadoc, "the rows are not sorted in each partition of the resulting Dataset."

The multi-column ZORDER/HILBERT paths in the same method do a real `RDD.sortBy`, so rows within each output partition (and therefore within each written file) are fully sorted by the curve ordinal. The single-column path leaves rows inside each partition in arbitrary order, so files produced by layout optimization on a single column are range-separated across files but unsorted within a file. That weakens page/row-group level min-max stats compared to the intended linear ordering.

This was surfaced while adding unit coverage in #19219: a test asserting ascending output for the single-column path failed deterministically in CI (rows came back in insertion order).

**To Reproduce**

Call `SpaceCurveSortingHelper.orderDataFrameByMappingValues(df, ZORDER, singletonList("c1"), 1)` on an unsorted frame and collect: the output preserves input order instead of ascending `c1` order.

**Expected behavior**

Either the single-column short-circuit should actually produce linear ordering (e.g. sort within partitions after the range repartition, matching the sort guarantee of the multi-column paths), or the comment/documentation should be corrected to state that only cross-partition range separation is provided.

**Environment Description**

* Hudi version : master

* Spark version : all supported

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.