Flink NBCC bulk insert may mix file groups across partitions
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Problem
When Flink bulk insert uses the bucket index with non-blocking concurrency control (NBCC), fixed bucket file IDs can repeat across partitions because the bucket ID is only unique within a partition.
The bulk-insert sort path currently orders records by file ID only, and `BucketBulkInsertWriterHelper` caches write handles by file ID only. Records from different partitions that share the same bucket ID can therefore be grouped together or reuse the wrong handle.
This affects both the regular bucket layout and the LSM bucket layout. The non-NBCC paths should retain their existing schema and sorting behavior to avoid unnecessary overhead.
### Expected behavior
In NBCC mode, bulk-insert records should be sorted and cached by the partition-aware file group identity, while non-NBCC behavior remains unchanged.
### Reproduction
Use a partitioned Flink table configured with:
- `operation = bulk_insert`
- a bucket index where the same bucket ID is present in multiple partitions
- `write.concurrency.mode = NON_BLOCKING_CONCURRENCY_CONTROL`
- bulk-insert input sorting enabled or disabled
Each partition should produce its own write status and file group even when the generated file IDs are identical.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.