apache / apache/fluss

Optimize the performance of Admin#listPartitionInfos(TablePath tablePath)

Open
#1,744 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

The method `Admin#listPartitionInfos(TablePath tablePath)` is currently called periodically by the partition discovery mechanism in `FlinkTableSource`. As a result, the number of API calls increases significantly as the number of subscribed consume jobs grows, leading to very frequent invocations. On the server side, listing partitions involves multiple requests to `ZooKeeper`, and for tables with a large number of partitions, this operation can take up to `10 seconds`. This issue will serve as the parent issue to address and optimize this performance problem.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at Admin#listPartitionInfos(TablePath tablePath) and trace its periodic use from FlinkTableSource's partition discovery mechanism. Then inspect the server-side partition listing and its ZooKeeper requests; done means reducing the frequent API and ZooKeeper work while improving performance for tables with many partitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.