apache / apache/hudi

Flink NBCC bulk insert may mix file groups across partitions

Open
#19,517 2 comments 0 reactions 1 assignee Claimed by @cshuo View on GitHub
engine:flink
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Problem

When Flink bulk insert uses the bucket index with non-blocking concurrency control (NBCC), fixed bucket file IDs can repeat across partitions because the bucket ID is only unique within a partition.

The bulk-insert sort path currently orders records by file ID only, and `BucketBulkInsertWriterHelper` caches write handles by file ID only. Records from different partitions that share the same bucket ID can therefore be grouped together or reuse the wrong handle.

This affects both the regular bucket layout and the LSM bucket layout. The non-NBCC paths should retain their existing schema and sorting behavior to avoid unnecessary overhead.

### Expected behavior

In NBCC mode, bulk-insert records should be sorted and cached by the partition-aware file group identity, while non-NBCC behavior remains unchanged.

### Reproduction

Use a partitioned Flink table configured with:

- `operation = bulk_insert`
- a bucket index where the same bucket ID is present in multiple partitions
- `write.concurrency.mode = NON_BLOCKING_CONCURRENCY_CONTROL`
- bulk-insert input sorting enabled or disabled

Each partition should produce its own write status and file group even when the generated file IDs are identical.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.