apache / apache/hudi

Expand NBCC to support other index types on Spark

Open
#16,651 2 comments 0 reactions 1 assignee Assigned to @yihua View on GitHub
from-jira priority:critical type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Right now, NBCC is only supported with simple bucket index on MOR.  We can consider relaxing this for use cases like ingestion with concurrent GDPR deletes to be supported by NBCC, using simple/global/RLI index on Spark.
{code:java}
if (writeConcurrencyMode == WriteConcurrencyMode.NON_BLOCKING_CONCURRENCY_CONTROL) {
checkArgument(
writeConfig.getTableType().equals(HoodieTableType.MERGE_ON_READ) && writeConfig.isSimpleBucketIndex(),
"Non-blocking concurrency control requires the MOR table with simple bucket index");
} {code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8300
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-6640
- Fix version(s):
- 1.1.0

---

## Comments

04/Oct/24 01:00;yihua;At high level, what needs to be done is to make sure the partitioner handles bucketing properly, not doing small file handling by merging inserts, updates, and deletes with existing base files.  The writer in NBCC only generates log files for updates/deletes (in some cases, we can still do conflict check on new file slices).
 
I'm thinking that inserts may not be identified across concurrent writers for regular index in NBCC if we relax the condition to support NBCC with simple/bloom/RLI index; but that is also a limitation of OCC, so we can note such limitation down and users should be aware of the limitation.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.