apache / apache/hudi-rs

feat(core): support multiple ordering fields end-to-end

Open
#600 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
279
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
40

Description

## Description

`hoodie.table.ordering.fields` supports comma-separated multiple fields (e.g., `ts,seq`) for lexicographic record ordering. The config parsing already handles this as a `List`, but the value is currently rejected at validation time (`UnsupportedValue`) because downstream code only handles a single ordering field.

## What needs to change

To support multiple ordering fields, the following code paths need updates:

### Data record merging (`merge/record_merger.rs`)
- `lexsort_to_indices` should build sort columns from all ordering fields instead of a single one
- Event time converter and extraction should handle multiple columns

### Ordering comparison (`merge/ordering.rs`)
- `process_batch_for_max_orderings` should extract and compare across all ordering fields
- `MaxOrderingInfo` event time comparison should be multi-column

### Record extraction (`record/mod.rs`)
- `create_event_time_ordering_converter` and `extract_event_time_ordering_values` should accept `&[String]` instead of `&str`

### Delete record handling
- Hudi delete records (`HoodieDeleteRecord.avsc`) only store a single `orderingVal`. Need to decide how multi-field ordering interacts with delete records — likely compare on the first field only for data-vs-delete comparisons, since the delete format is a Hudi spec constraint.

### Config validation (`config/table.rs`)
- Remove the `UnsupportedValue` guard for multiple ordering fields in `parse_value`

## Context

Introduced in #598 — config alias support and `OrderingFields` rename.

Contributor guide

Open the contributing guide

Research direction

Start with config/table.rs to trace the current UnsupportedValue validation, then read merge/record_merger.rs, merge/ordering.rs, and record/mod.rs at the named functions. Review HoodieDeleteRecord.avsc before deciding how delete records participate in multi-field comparisons. Done means multiple ordering fields pass validation and are handled through merging, ordering comparison, extraction, and delete-record behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.