apache / apache/hudi

[SUPPORT] Error when running a pipeline after an interrupt

Open
#9,518 8 comments 0 reactions 0 assignees View on GitHub
area:writer priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Currently, we have a pipeline with approximately 2 billion records and 95 columns that runs every day. Yesterday, at the time of execution, there was an intermittence in the EKS that we have to execute the pipeline (we don't know in which part of the execution, if it was in the inflight part or already in the commit). With that, when we tried to perform a next execution, we had the error below:

![Captura de Tela 2023-08-24 às 10 13 32](https://github.com/apache/hudi/assets/89396112/23ab7538-6948-4749-ab23-14c4f0bcfb2f)

We tried everything: running low volume, high volume, changing spark resources, and we always got the same error. We also tried to make a savepoint, but when performing the restore we also had errors. In that case, how to proceed?

OBS: I checked the entire log and there are no other errors, just this one, repeating itself several times

**To Reproduce**

Steps to reproduce the behavior:

1. Start a pipeline with Hudi 0.12.0 running on EKS
2. Stop the pipeline abruptly
3. Error

**Expected behavior**

Despite the interruption, I expected the pipeline to start working again.

**Environment Description**

* EMR on EKS

* Hudi version : 0.12.2

* Spark version : 3.2.1

* Hive version : -

* Hadoop version : -

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : Running on EKS

Contributor guide

No contributing guide indexed for this repository

Research direction

Use the reported reproduction steps with Hudi 0.12.2, Spark 3.2.1, EMR on EKS, and S3, starting with the full repeated error from the attached log or screenshot. Determine whether an abrupt interruption leaves the pipeline unable to restart or restore, and document a reproducible recovery path or the missing diagnostic information.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, spark
Domain
cloud, data-engineering, distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.