hashicorp / hashicorp/consul

RPC endpoints do not support cancellation

Open
#10,206 0 comments 0 reactions 0 assignees View on GitHub
theme/reliability type/enhancement type/umbrella-☂️
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

The RPC server uses [net/rpc](https://golang.org/pkg/net/rpc/) which does not support cancellation of requests (and also happens to have been ~deprecated~ [frozen since 2016](https://github.com/golang/go/issues/16844)).

For requests that are consistently short lived, this lack of cancellation is not a significant problem. However, many of the RPC endpoints support [blocking queries](https://www.consul.io/api-docs/features/blocking) (long polling) which are expected to be long living requests (up to some timeout, defaults to 10 minutes). If the client performing the request (via the CLI, or HTTP API) attempts to cancel the request, the RPC endpoint keeps blocking until the timeout.

The symptoms of this problem are not obvious. It can look like a file descriptor or goroutine leak because the resources used by the request are held until the timeout. In extreme cases, where a client frequently cancels requests and starts new ones, this could waste a lot of resources on the server.

Some options for addressing this problem are:
1. #6366 - replace all blocking queries with streaming. Streaming is built on `gRPC` which supports cancellation.
2. replace `net/rpc` with `gRPC` while continuing to use blocking queries (less work overall than using streaming)
3. https://github.com/hashicorp/consul/issues/9647#issuecomment-823571538 - replace `net/rpc` with [keegancsmith/rpc](https://github.com/keegancsmith/rpc), however based on the initial analysis this doesn't seem like it will work with our implementation.

Issues that are likely caused by this problem:
* #9647
* #10193

Contributor guide

Open the contributing guide

Research direction

Start with the RPC server and the blocking-query endpoints used by the CLI and HTTP API, then compare the listed replacement options, including gRPC and streaming. Done means a client cancellation stops the long-running request and releases its resources instead of waiting for the timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
api, backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.