Expand NBCC to support other index types on Spark
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Right now, NBCC is only supported with simple bucket index on MOR. We can consider relaxing this for use cases like ingestion with concurrent GDPR deletes to be supported by NBCC, using simple/global/RLI index on Spark.
{code:java}
if (writeConcurrencyMode == WriteConcurrencyMode.NON_BLOCKING_CONCURRENCY_CONTROL) {
checkArgument(
writeConfig.getTableType().equals(HoodieTableType.MERGE_ON_READ) && writeConfig.isSimpleBucketIndex(),
"Non-blocking concurrency control requires the MOR table with simple bucket index");
} {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8300
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-6640
- Fix version(s):
- 1.1.0
---
## Comments
04/Oct/24 01:00;yihua;At high level, what needs to be done is to make sure the partitioner handles bucketing properly, not doing small file handling by merging inserts, updates, and deletes with existing base files. The writer in NBCC only generates log files for updates/deletes (in some cases, we can still do conflict check on new file slices).
I'm thinking that inserts may not be identified across concurrent writers for regular index in NBCC if we relax the condition to support NBCC with simple/bloom/RLI index; but that is also a limitation of OCC, so we can note such limitation down and users should be aware of the limitation.;;;
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.