apache / apache/fluss

Handle individual table failures without failing the entire job

Open
#2,360 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

**Description**
Currently, the Tiering Service processes multiple tables for data tiering. However, if a task for a specific table fails due to non-recoverable reasons (e.g., AccessDeniedException for remote storage, missing Paimon Catalog permissions, or metadata corruption), the entire Tiering Service job might crash or enter a restart loop.

Consistent with our implementation in the Committer node, we should improve the fault tolerance of the Tiering Service.

**Proposed Changes**
The Tiering Service should be decoupled from individual table task failures:

Isolation of Failures: If a tiering task for Table A fails due to an exception (especially permission-related or configuration-related), the service should catch the exception and prevent it from propagating to the main thread.

Table Status Marking: Mark the specific table as "Failed" or "Error" in memory (and potentially log the state).

Continue with Next Table: The service should continue processing the next available tables (Table B, Table C, etc.) in the queue instead of stopping the entire job.

The `TieringReader` should send to `TieringEnumerator` and the `TieringEnumerator` should notify this to all `TieringReader` and `Committer` so that they can clear their status for this failed table.

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the TieringReader, TieringEnumerator, and Committer entry points, then compare their failure handling with the existing Committer implementation. Trace how a failed table task should be reported, marked, and cleared while later tables continue processing; done means one table failure no longer stops the Tiering Service and the affected status is communicated to the relevant components.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.