[Bug] New Partiions created will cause Flink job failover when partition discovery is enabled while scanning
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.
### Fluss version
0.5.0
### Minimal reproduce step
Create a partitioned table and a Flink job to subscribe it.. After new partitions is created, it may well throw
```
Caused by: com.alibaba.fluss.exception.FlussRuntimeException: Leader not found for table bucket: TableBucket{tableId=0, partitionId=60, bucket=16}
at com.alibaba.fluss.client.metadata.MetadataUpdater.leaderFor(MetadataUpdater.java:122)
at com.alibaba.fluss.client.admin.FlussAdmin.prepareListOffsetsRequests(FlussAdmin.java:295)
at com.alibaba.fluss.client.admin.FlussAdmin.listOffsets(FlussAdmin.java:253)
at com.alibaba.fluss.connector.flink.source.enumerator.initializer.BucketOffsetsRetrieverImpl.listOffsets(BucketOffsetsRetrieverImpl.java:72)
at com.alibaba.fluss.connector.flink.source.enumerator.initializer.BucketOffsetsRetrieverImpl.latestOffsets(BucketOffsetsRetrieverImpl.java:49)
at com.alibaba.fluss.connector.flink.source.enumerator.initializer.LatestOffsetsInitializer.getBucketOffsets(LatestOffsetsInitializer.java:38)
at com.alibaba.fluss.connector.flink.source.enumerator.FlinkSourceEnumerator.getLogSplit(FlinkSourceEnumerator.java:449)
at com.alibaba.fluss.connector.flink.source.enumerator.FlinkSourceEnumerator.initLogTablePartitionSplits(FlinkSourceEnumerator.java:356)
at com.alibaba.fluss.connector.flink.source.enumerator.FlinkSourceEnumerator.initPartitionedSplits(FlinkSourceEnumerator.java:348)
at com.alibaba.fluss.connector.flink.source.enumerator.FlinkSourceEnumerator.lambda$checkPartitionChanges$2(FlinkSourceEnumerator.java:296)
at org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$2(ExecutorNotifier.java:80)
```
It'll cause job failover
### What doesn't meet your expectations?
It's in the expectation that throws an exception since the leader may well still not be elected although partition has been created...Although it'll running normally after failover, we'd better to handle this cause to avoid failover again and again in every partition created.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.