CDC/Incremental queries with start instant archived (in history) produce incorrect result
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
How to reproduce:
1. Use the CDC query example in the quick start: [https://hudi.apache.org/docs/0.15.0/quick-start-guide]
2. Use the following setting to create archived commits, and try more updates queries:
{code:java}
val opts: Map[String, String] = Map(
"hoodie.archive.automatic" -> "true",
"hoodie.keep.max.commits" -> "3",
"hoodie.keep.min.commits" -> "2",
"hoodie.clean.automatic" -> "clean",
"hoodie.clean.policy" -> "KEEP_LATEST_COMMITS",
"hoodie.clean.trigger.max.commits" -> "1",
"hoodie.clean.commits.retained" -> "3")
val updatesDf = spark.read.format("hudi").load(basePath).withColumn("fare", col("fare") * 2)
updatesDf.write.format("hudi").
option("hoodie.datasource.write.operation", "upsert").
option("hoodie.datasource.write.partitionpath.field", "city").
option("hoodie.table.cdc.enabled", "true").
option("hoodie.table.name", tableName).
options(opts).
mode(Append).
save(basePath)
{code}
Meanwhile, the incremental query neither produce the correct result.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8856
- Type: Bug
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.