apache / apache/fluss

FlinkSink write to primaryKey table may block forever When upgrading the Fluss cluster

Open
#742 1 comment 0 reactions 0 assignees View on GitHub
component=client component=connector/flink
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Fluss version

main (development)

### Please describe the bug 🐞

FlinkSink write to primaryKey table may block forever When upgrading the Fluss cluster. Assuming this primaryKey table currently has 128 buckets, and the parallelism of the `FlinkWriter` is also set to 128. Additionally, after a `tabletServer` restarts, its assigned IP address will change (due to pod deployment). Because of the existing `bucket_key shuffle`, there may be cases where one write task only handles a single hash key. For instance, if some of our write tasks are processing data solely from tabletServer-0, and I restart the cluster by continuously killing nodes until finally killing tabletServer-0, all the IP addresses in the client's metadata will become invalid. When the task found that tabletServer-0 is unreachable and attempt to retry updating metadata, it will fail because all the IPs in the client's metadata are incorrect.

### Solution

Maybe we need to use bootstrap server to update metadata.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the FlinkSink entry point and trace the bucket_key shuffle and metadata-retry path during tablet-server restarts. Reproduce the 128-bucket, 128-writer upgrade scenario and inspect how bootstrap servers could refresh metadata after tablet-server IPs change; done means writes no longer block forever and retries recover.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.