apache / apache/hudi

Allow retries on marker file creation requests

Open
#18,072 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
Cleaner failures due to marker retries

**What you expected:**
Retries to suceed for markers

**Fix needed**
When using the remote marker files, if the request times out but the marker is created then the retry will fail. This should succeed. We need to add some request ID or other tracker that allows us to make idempotent retries without creating other risks.

### Environment

**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue does not identify any files, tests, or entry points. Start by locating the remote marker-file creation request and its retry handling, then determine how a timed-out request can be retried safely and how completion should be verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.