RPC endpoints do not support cancellation
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
The RPC server uses [net/rpc](https://golang.org/pkg/net/rpc/) which does not support cancellation of requests (and also happens to have been ~deprecated~ [frozen since 2016](https://github.com/golang/go/issues/16844)).
For requests that are consistently short lived, this lack of cancellation is not a significant problem. However, many of the RPC endpoints support [blocking queries](https://www.consul.io/api-docs/features/blocking) (long polling) which are expected to be long living requests (up to some timeout, defaults to 10 minutes). If the client performing the request (via the CLI, or HTTP API) attempts to cancel the request, the RPC endpoint keeps blocking until the timeout.
The symptoms of this problem are not obvious. It can look like a file descriptor or goroutine leak because the resources used by the request are held until the timeout. In extreme cases, where a client frequently cancels requests and starts new ones, this could waste a lot of resources on the server.
Some options for addressing this problem are:
1. #6366 - replace all blocking queries with streaming. Streaming is built on `gRPC` which supports cancellation.
2. replace `net/rpc` with `gRPC` while continuing to use blocking queries (less work overall than using streaming)
3. https://github.com/hashicorp/consul/issues/9647#issuecomment-823571538 - replace `net/rpc` with [keegancsmith/rpc](https://github.com/keegancsmith/rpc), however based on the initial analysis this doesn't seem like it will work with our implementation.
Issues that are likely caused by this problem:
* #9647
* #10193
Contributor guide
Research direction
Start with the RPC server and the blocking-query endpoints used by the CLI and HTTP API, then compare the listed replacement options, including gRPC and streaming. Done means a client cancellation stops the long-running request and releases its resources instead of waiting for the timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc
- Domain
- api, backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100