apache / apache/hudi

Implement Spark bootstrap for IVF clustering and MDT posting generation

Open
#19,099 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Parent: #19094
Related: #18676

## Summary
Implement Spark bootstrap for IVF clustering and MDT posting generation.

## Scope
- centroid training
- vector assignment to centroids
- posting generation
- quantizer and manifest rows
- generation write path

## Acceptance Criteria
- index bootstrap produces a complete generation
- bootstrap output is persisted to MDT
- bootstrap logic is separate from read-path logic

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading parent issue #19094 and related issue #18676, since the payload names no files, tests, or code entry points. Trace the requested centroid training, assignment, posting, quantizer, manifest, and generation write-path components, and consider the work complete when a full index generation is produced and persisted to MDT with bootstrap logic separate from read-path logic.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.