apache / apache/pulsar

PIP-183: Reduce unnecessary REST call in broker

Open
#16,422 4 comments 0 reactions 0 assignees View on GitHub
Stale type/PIP
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

## Motivation

The design of admin API now is such that: when handle a partitioned topic request, the broker will query the topic's partition meta, and then use the internal admin client to query all the non-partitioned topics (I.e. the suffix of the topic name is `-partition-`),
even if the non-partitioned topic is owned by the broker, which will cause unnecessary REST call in the broker.

we can call the methods directlly, who handle the non-partitioned topic, to reduce the unnecessary REST call.

## Goal

* Try to call the methods directlly if the non-partitioned topic is owned by the broker

## Implementation

* We need to check all the place where `org.apache.pulsar.broker.PulsarService#getAdminClient` is invoked in `org.apache.pulsar.broker.admin.impl.PersistentTopicsBase`
* take `internalGetPartitionedStats` for example:

* Original:
```
for (int i = 0; i < partitionMetadata.partitions; i++) {
try {
topicStatsFutureList
.add(pulsar().getAdminClient().topics().getStatsAsync(
(topicName.getPartition(i).toString()), getPreciseBacklog, subscriptionBacklogSize,
getEarliestTimeInBacklog));
} catch (PulsarServerException e) {
asyncResponse.resume(new RestException(e));
return;
}
}
```

* Suggest to do like this:
```
for (int i = 0; i < partitionMetadata.partitions; i++) {
TopicName topicNamePartition = topicName.getPartition(i);
topicStatsFutureList.add(
pulsar().getNamespaceService().isServiceUnitOwnedAsync(topicName)
.thenCompose(owned -> {
if (owned) {
// local call
return getTopicReferenceAsync(topicNamePartition)
.thenCompose(topic ->
topic.asyncGetStats(getPreciseBacklog, subscriptionBacklogSize,
getEarliestTimeInBacklog));
} else {
// call from admin client
try {
pulsar().getAdminClient().topics().getStatsAsync(topicNamePartition.toString()),
getPreciseBacklog, subscriptionBacklogSize, getEarliestTimeInBacklog)
} catch (PulsarServerException e) {
throw new RestException(e);
}
}
})
);
```

Contributor guide

Open the contributing guide

Research direction

Start in org.apache.pulsar.broker.admin.impl.PersistentTopicsBase and inspect every use of PulsarService#getAdminClient, beginning with internalGetPartitionedStats. Trace how ownership and local topic references are handled, then verify that locally owned non-partitioned topics avoid the REST call while remote topics retain the admin-client path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.