apache / apache/fluss

[tiering] State machine race: first tiering round fails with 'Scheduled to Tiered is not a valid state change'

Open
#3,672 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

## Summary

The first tiering round for a newly created datalake-enabled table fails with a state machine validation error. Subsequent rounds succeed.

## Error

`
ERROR org.apache.fluss.server.coordinator.LakeTableTieringManager - Fail to change state for table 1 from Scheduled to Tiered as it's not a valid state change.
ERROR org.apache.fluss.server.coordinator.LakeTableTieringManager - Fail to change state for table 1 from Scheduled to Scheduled as it's not a valid state change.
`

## Expected Behavior

The state machine should transition Scheduled Pending Tiering Tiered without errors on the first tiering round.

## Actual Behavior

The tiering job completes its work (S3 write succeeds, `storeLakeTableOffsetsFile()` succeeds) but the Coordinator rejects the state transition because the table is still in \Scheduled\ state when the tiering result arrives. The expected intermediate transitions (Scheduled Pending Tiering) haven't occurred yet.

The second tiering round works fine by then the state machine has caught up.

## Steps to Reproduce

1. Create a table with 'table.datalake.enabled' = 'true', 'table.datalake.freshness' = '60s'
2. Insert a small amount of data (e.g., 3 rows)
3. Start the tiering service
4. Observe Coordinator logs within the first freshness interval

## Environment

- Fluss: built from main (post-PR #3550)
- Flink: 1.20.1
- Iceberg: 1.10.1 with Glue catalog
- Infrastructure: ECS Fargate (single TabletServer, single bucket)

## Analysis

Likely a timing issue: with a small dataset and fast S3 writes, the tiering job completes its first round before `LakeTableTieringManager` has transitioned the table through the intermediate states. The state machine expects the sequence Scheduled Pending Tiering Tiered, but the tiering completion arrives while still in Scheduled.

This may be more likely with:
- Small datasets (fast tiering)
- Fast network to S3
- Short freshness intervals

## Impact

- First tiering round's snapshot is lost (not committed to Iceberg metadata)
- Subsequent rounds work correctly
- Data is eventually tiered, but there's a one-round delay on fresh tables

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the LakeTableTieringManager entry point and the Coordinator state-transition handling described in the error. Reproduce with a newly created datalake-enabled table, three rows, and a 60-second freshness interval, then trace the first tiering round. Done means the first round completes without invalid Scheduled transitions and its snapshot is committed to Iceberg metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.