[FEATURE] Add rpc queued time and rpc process time.
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
When I perform a stress test on the cluster, tasks occasionally encounter errors like this:
```
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [closed=[CANCELLED], committed=[remote_addr=xxx/xxx:xxx]]
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
```
> We can increase `rss.rpc.executor.size` to reduce the probability of problems.
In this time, I will see metrics `grpc_server_executor_blocking_queue_size` like this:
This may mean that rpc requests are not processed in a timely manner. But there are no indicators that directly prove this. So we should add rpc queued time and rpc process time.
### Motivation
_No response_
### Describe the solution
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Contributor guide
Assessment
This issue has not been assessed yet.