[Vector Index] (Tracking) RaBitQ, read-path pruning, write path, maintenance

Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).

## Scope

**Tracking issue only.** These items are explicitly **out of scope** for the milestone-1 sub-issues (1–7). They live here so they don't get lost — each will be broken out into its own sub-issue once milestone 1 lands.

## Deferred work

- **RaBitQ quantization**: replace the raw `array` payload with packed binary codes + optional norm scalar (see `RaBitQEncoder.java`, `VectorQuantizer.java` in the [design PR](https://github.com/chrevanthreddy/hudi/pull/1)).
- **Generation manifest & quantizer record**: `__manifest__` / `__centroids__` / `__quantizer__` rows for atomic generation activation.
- **Read-path pruning**: `VectorIndexPruner`, `VectorIndexMdtSearchUtils`, `RaBitQApproxDistanceUDF`, `VectorIndexSupport.scala`.
- **Write path**: assign incoming records to clusters at write time, write tombstones for deletes (RFC-104 write-path doc).
- **Maintenance**: cluster-imbalance / centroid-drift detection, LIRE-style incremental rebalancing, generation rebuild.
- **Flink and Java engine support** (Spark-first stays as Spark-only in milestone 1).

## Action

Leave this issue open; close once milestone 1 (sub-issues 1–7) is merged and follow-up sub-issues are filed for each item above.

This is a tracking issue rather than an implementation task. Read RFC-104 and the linked design PR first, then review the named entry points such as RaBitQEncoder.java, VectorQuantizer.java, VectorIndexPruner, and VectorIndexSupport.scala. Work should wait for milestone 1 and be split into follow-up sub-issues; close this issue only after those sub-issues are filed and milestone 1 is merged.

Written by the indexing model from the issue text.