apache / apache/pulsar

PIP-223: Add metrics for all Rest Endpoints

Open
#18,560 4 comments 0 reactions 1 assignee Claimed by @tjiuming View on GitHub
Stale type/PIP
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Motivation

Currently, we have [WebExecutor metrics](https://pulsar.apache.org/docs/next/reference-metrics/#web-service-executor-metrics) and [Jetty metrics](https://pulsar.apache.org/docs/2.10.x/reference-metrics#jetty-metrics) in the pulsar broker.

The granularity of these metrics is relatively large, we can only know the metrics of the Server dimension, but we cannot know the metrics of each specific endpoint.

This can not provide more help when finding and troubleshooting problems.

For the purpose of solve the above questions, it's is better to introduce per-endpoint metrics.

### Goal

Provide per-endpoint metrics.

### API Changes

_No response_

### Implementation

#### 1. Configuration

Add the following configuration to [ServiceConfiguration.java](https://github.com/apache/pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java), used to control whether to open all Rest-Endpoints monitoring, which default value is false.

```
private boolean exposePerRestEndpointMetricsInPrometheus = false;
```

#### 2. Implementation

Pulsar using [Jersey](https://github.com/eclipse-ee4j/jersey) framework to provide RestAPI service, and Jersey provided [ContainerRequestFilter](https://github.com/javaee/jax-rs-api/blob/master/jaxrs-api/src/main/java/javax/ws/rs/container/ContainerRequestFilter.java), [ContainerResponseFilter](https://github.com/javaee/jax-rs-api/blob/master/jaxrs-api/src/main/java/javax/ws/rs/container/ContainerResponseFilter.java).

We can use these two filters to monitoring each endpoint's metrics:
```
public class RestEndpointMetricsFilter implements ContainerResponseFilter, ContainerRequestFilter {

private static final String REQUEST_START_TIME = "requestStartTime";

@Override
public void filter(ContainerRequestContext req, ContainerResponseContext resp) throws IOException {
String path = req.getUriInfo().getPath();
String method = req.getMethod();
long requestStartTime = req.getProperty(REQUEST_START_TIME);
// ignore....
}

@Override
public void filter(ContainerRequestContext req) throws IOException {
// Set the request start time into properties.
req.setProperty(REQUEST_START_TIME, System.currentTimeMillis());
}
```

But there is a problem, for an Rest endpoint
```
@Path("/user")
public class Users {

@Path("/{id}/query")
public User queryById(@PathParam("id") long userId) {
return xxx;
}
}
```
the request paths can be `/user/1111/query`,`/user/2222/query`,`/user/3333/query` and etc. It will lead to tooooo many time series and may even lead to downtime of Prometheus.

So we have to group these paths to `/user/:id/query`, and this will be a very good optimization.

Fortunately, we can get the method of providing services through `req.getUriInfo()`, and get the content of `@Path` through reflection, and we will cache the Path to prevent cost too much from reflection:
```
@Override
public void filter(ContainerRequestContext req, ContainerResponseContext resp) throws IOException {
UriRoutingContext info = (UriRoutingContext) req.getUriInfo();
ResourceMethod rm = info.getMatchedResourceMethod();
Invocable inv = method.getInvocable();
// The endpoint handler class
Class handlingClass = inv.getHandler().getHandlerClass();
// The endpoint handler method
Method handlingMethod = inv.getHandlingMethod();
Path parent = handlingClass.getDeclaredAnnotation(Path.class);
Path child = handlingMethod.getDeclaredAnnotation(Path.class);

// The base path
String parent0 = parent == null ? "" : parent.value();
// The method path
String child0 = child == null ? "" : child.value();

String path = parent0.endsWith("/") || child0.startsWith("/")
? parent0 + child0 : parent0 + "/" + child0;
}
```
After all, we can monitor each endpoint's metrics.

#### 3. Metrics
| Name | Type | Description |
| ----------------------------------------- | ----------- | ---------------------------------------------------- |
| pulsar_broker_rest_endpoint_latency_ms | Histogram | The latency of the request execution. |
| pulsar_broker_rest_endpoint_failed | Counter | The number of times this Endpoint failed(The http response code >= 400.) to execute. |

##### a. Labels
The 2 metrics has the following labels:
i. path: The HTTP request path
ii. method: The HTTP request method
iii. code: The HTTP response code

### Alternatives

_No response_

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.