apache / apache/fluss

[Bug] The time out for UpdateMetadata request is too short to cause job failover

Open
#311 1 comment 0 reactions 1 assignee Assigned to @wuchong View on GitHub
component=client
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Fluss version

main

### Minimal reproduce step

Currently, `sendMetadataRequestAndRebuildCluster` will have a timeout of 3s. After 3s, the future will be completed without updating metadata.

### What doesn't meet your expectations?

In my case, a partitioned table with 512 buckets, 512 parallelism for flink sink, it'll be timeout easily and then cause sink job fail..

First, write to a partition, it will try to update the metadata in method `checkAndUpdatePartitionMetadata` to fetch the partition's metadata. If it timeout, the metadata won't be updated in client, and it will then throw PartitionNotExists exception althogth the Partition does exist.

For my case, a time out of 60s works....

### Anything else?

I can see we need to introduce a request time out mechanism to avoid a request to hang out forever.. But for updating metadata request, it should throw Timeout exception instead of just log it to enable caller to decide retry or fail directly..

For example, when creating `FlussTable`, it'll try to fetch the metadata of the table in [metadataUpdater.checkAndUpdateTableMetadata(Collections.singleton(tablePath))](https://github.com/alibaba/fluss/blob/92c6d23bd25b9de61e05859e18c5b267c81f80db/fluss-client/src/main/java/com/alibaba/fluss/client/table/FlussTable.java#L145). If the metadata is timeout, the metadata can't be updated and cause it to throw `table not found in cluster` exception although the table does exist... At least, it should throw timeout exception instead of `table not found in cluster` exception which is really confused.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.