apache / apache/hudi

CDC/Incremental queries with start instant archived (in history) produce incorrect result

Open
#16,777 1 comment 0 reactions 1 assignee Assigned to @linliu-code View on GitHub
from-jira priority:blocker type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

How to reproduce:

1. Use the CDC query example in the quick start: [https://hudi.apache.org/docs/0.15.0/quick-start-guide]

2. Use the following setting to create archived commits, and try more updates queries:
{code:java}
val opts: Map[String, String] = Map(
  "hoodie.archive.automatic" -> "true",
  "hoodie.keep.max.commits" -> "3",
  "hoodie.keep.min.commits" -> "2",
  "hoodie.clean.automatic" -> "clean",
  "hoodie.clean.policy" -> "KEEP_LATEST_COMMITS",
  "hoodie.clean.trigger.max.commits" -> "1",
  "hoodie.clean.commits.retained" -> "3") 
val updatesDf = spark.read.format("hudi").load(basePath).withColumn("fare", col("fare") * 2)
updatesDf.write.format("hudi").
  option("hoodie.datasource.write.operation", "upsert").
  option("hoodie.datasource.write.partitionpath.field", "city").
  option("hoodie.table.cdc.enabled", "true").
  option("hoodie.table.name", tableName).
  options(opts).
  mode(Append).
  save(basePath)

{code}
 

Meanwhile, the incremental query neither produce the correct result.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8856
- Type: Bug
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.