apache / apache/fluss

Recovery failure when the tablet server shuts down due to 'No space left on device'

Open
#3,224 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

0.9.0 (latest release)

### Please describe the bug 🐞

I found that recovery fails when the tablet server shuts down due to a 'No space left on device' error. The failure logs are as follows:

Image

However, after dropping the corresponding partitions by fluss admin, partition recovery is still not skipped. This indicates that the behavior described in #3223 can be further improved.

In addition, we should prevent writes once the disk usage reaches a high-capacity watermark to avoid potential data loss.

### Solution

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the recovery behavior described in issue #3223 and the tablet-server shutdown logs for the disk-full case. Verify why dropping the corresponding partitions through Fluss admin does not skip recovery, and define completion as correct recovery handling plus writes being prevented at the high-capacity watermark.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.