apache / apache/hudi

[Vector Index] File-group mapping function for cluster-to-file-group routing

Open
#18,852 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).

## Scope

Records belonging to the same cluster must land in the same contiguous bucket of MDT file groups (cluster = a folder containing N files). This sub-task adds the mapping function used by the MDT writer.

## Tasks

- Add `getVectorKeyToFileGroupMappingFunction(numClusters, fgPerCluster)` in `hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java`.
- Key encoding: prefix the record key with the cluster ID, e.g. `C|`. Allows prefix scans per cluster at read time.
- Mapping: `fileGroupIndex = (clusterId * fgPerCluster) + (hash(recordKey) % fgPerCluster)`.
- Override `getFileGroupMappingFunction(HoodieIndexVersion)` on the `VECTOR_INDEX` enum in `MetadataPartitionType` so MDT routes records to the right file group.

## Tests

- Unit test: insert many synthetic `(recordKey, clusterId)` tuples; assert all records for cluster `c` land in file groups `[c*fgPerCluster, (c+1)*fgPerCluster)`.
- Unit test: varying `fgPerCluster` (1, 4, 16) — distribution of records within a cluster is roughly uniform across that cluster's file groups.

## Depends on

- Sub-issue 1 (partition type registration)

## Out of scope

Actual writing into the file groups — that happens in sub-issue 5.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading getVectorKeyToFileGroupMappingFunction in hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java and the VECTOR_INDEX entry in MetadataPartitionType. Run the relevant metadata unit tests, then add coverage for cluster ranges and roughly uniform distribution with fgPerCluster values 1, 4, and 16. Done means keys use the specified cluster prefix and map only within each cluster's file-group range.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.