[Vector Index] File-group mapping function for cluster-to-file-group routing
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).
## Scope
Records belonging to the same cluster must land in the same contiguous bucket of MDT file groups (cluster = a folder containing N files). This sub-task adds the mapping function used by the MDT writer.
## Tasks
- Add `getVectorKeyToFileGroupMappingFunction(numClusters, fgPerCluster)` in `hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java`.
- Key encoding: prefix the record key with the cluster ID, e.g. `C|`. Allows prefix scans per cluster at read time.
- Mapping: `fileGroupIndex = (clusterId * fgPerCluster) + (hash(recordKey) % fgPerCluster)`.
- Override `getFileGroupMappingFunction(HoodieIndexVersion)` on the `VECTOR_INDEX` enum in `MetadataPartitionType` so MDT routes records to the right file group.
## Tests
- Unit test: insert many synthetic `(recordKey, clusterId)` tuples; assert all records for cluster `c` land in file groups `[c*fgPerCluster, (c+1)*fgPerCluster)`.
- Unit test: varying `fgPerCluster` (1, 4, 16) — distribution of records within a cluster is roughly uniform across that cluster's file groups.
## Depends on
- Sub-issue 1 (partition type registration)
## Out of scope
Actual writing into the file groups — that happens in sub-issue 5.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading getVectorKeyToFileGroupMappingFunction in hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java and the VECTOR_INDEX entry in MetadataPartitionType. Run the relevant metadata unit tests, then add coverage for cluster ranges and roughly uniform distribution with fgPerCluster values 1, 4, and 16. Done means keys use the specified cluster prefix and map only within each cluster's file-group range.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100