apache / apache/hudi

Found commits after time :20230220161017756, please rollback greater commits first

Open
#8,025 9 comments 0 reactions 0 assignees View on GitHub
area:table-service
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

Hi,
Our streaming job is failing with Found commits after time :20230220161017756, please rollback greater commits first,

We tried to rollback commits with the command commit rollback --commit commit_num but we are facing the below exception
Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant:[20230221130513323__deltacommit__COMPLETED]
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.(BaseRollbackActionExecutor.java:93)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.(BaseRollbackActionExecutor.java:73)
at org.apache.hudi.table.action.rollback.MergeOnReadRollbackActionExecutor.(MergeOnReadRollbackActionExecutor.java:48)
at org.apache.hudi.table.HoodieSparkMergeOnReadTable.rollback(HoodieSparkMergeOnReadTable.java:170)
at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:766)
... 15 more

Our Hudi configs are

DataSourceWriteOptions.TABLE_TYPE.key() -> DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL,
DataSourceWriteOptions.RECORDKEY_FIELD.key() -> "guid",
DataSourceWriteOptions.PARTITIONPATH_FIELD.key() -> "collectionName",
DataSourceWriteOptions.PRECOMBINE_FIELD.key() -> "operationTime",
HoodieCompactionConfig.INLINE_COMPACT_TRIGGER_STRATEGY.key() -> CompactionTriggerStrategy.TIME_ELAPSED.name,
HoodieCompactionConfig.INLINE_COMPACT_TIME_DELTA_SECONDS.key() -> String.valueOf(60 * 60),
HoodieCompactionConfig.CLEANER_POLICY.key() -> HoodieCleaningPolicy.KEEP_LATEST_COMMITS.name(),
HoodieCompactionConfig.CLEANER_COMMITS_RETAINED.key() -> "624",
HoodieCompactionConfig.MIN_COMMITS_TO_KEEP.key() -> "625",
HoodieCompactionConfig.MAX_COMMITS_TO_KEEP.key() -> "648",
HoodieCompactionConfig.ASYNC_CLEAN.key() -> "false",
HoodieCompactionConfig.INLINE_COMPACT.key() -> "true",
HoodieMetricsConfig.TURN_METRICS_ON.key() -> "true",
HoodieMetricsConfig.METRICS_REPORTER_TYPE_VALUE.key() -> MetricsReporterType.DATADOG.name(),
HoodieMetricsDatadogConfig.API_SITE_VALUE.key() -> "US",
HoodieMetricsDatadogConfig.METRIC_PREFIX_VALUE.key() -> "tacticalnovusingest.hudi",
HoodieMetricsDatadogConfig.API_KEY_SUPPLIER.key() -> "com.tr.indigo.tacticalnovusingest.utils.DatadogKeySupplier",
HoodieMetadataConfig.ENABLE.key() -> "false",
HoodieWriteConfig.ROLLBACK_USING_MARKERS_ENABLE.key() -> "false",
Please help what caused this issue
Help us how to resolve this

**Environment Description** AWS

* Hudi version : 0.11.1

* Spark version : 3.1.2

* Hive version : NA

* Hadoop version :3

* Storage (HDFS/S3/GCS..) :S3

* Running on Docker? (yes/no) :no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the rollback path through BaseHoodieWriteClient.rollback, HoodieSparkMergeOnReadTable.rollback, and BaseRollbackActionExecutor, using the reported completed instant and rollback configuration as the reproducer. Check existing rollback tests for completed instants and marker-based strategies. Done means the cause is reproduced and a verified resolution or clearly documented limitation is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.