FlinkSink write to primaryKey table may block forever When upgrading the Fluss cluster
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.
### Fluss version
main (development)
### Please describe the bug 🐞
FlinkSink write to primaryKey table may block forever When upgrading the Fluss cluster. Assuming this primaryKey table currently has 128 buckets, and the parallelism of the `FlinkWriter` is also set to 128. Additionally, after a `tabletServer` restarts, its assigned IP address will change (due to pod deployment). Because of the existing `bucket_key shuffle`, there may be cases where one write task only handles a single hash key. For instance, if some of our write tasks are processing data solely from tabletServer-0, and I restart the cluster by continuously killing nodes until finally killing tabletServer-0, all the IP addresses in the client's metadata will become invalid. When the task found that tabletServer-0 is unreachable and attempt to retry updating metadata, it will fail because all the IPs in the client's metadata are incorrect.
### Solution
Maybe we need to use bootstrap server to update metadata.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the FlinkSink entry point and trace the bucket_key shuffle and metadata-retry path during tablet-server restarts. Reproduce the 128-bucket, 128-writer upgrade scenario and inspect how bootstrap servers could refresh metadata after tablet-server IPs change; done means writes no longer block forever and retries recover.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100