hashicorp / hashicorp/consul

Multi-tenant Isolation For Consul Control Plane

Open
#13,769 1 comment 8 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Context

We are running Consul in a multi-tenant fashion. We have large consul cluster(s) with many consul agents (>>10k). Many application teams registered a large number of services into Consul. Teams not only use Consul as a service catalog, some teams also use consul’s KV functionalities. As Consul usage grows organically, more load and variation of usage patterns grows significantly. Consul operators cannot effectively enforce best practices in reliability and scalability context for consul usage across all the teams. This impacts the reliability of our consul deployment significantly.

As our past incidents have shown, a single bad tenant (e.g. one service owned by one team) was able to bring down our consul cluster. Many services depend on Consul on their critical path. This in turn caused large scale outages across many services.

#### Feature Description

Support multi-tenant isolation natively in Consul. Facilitate running large consul deployments reliably in multi-tenant environment. Prevent one or few bad tenants (e.g. misbehaving, DDoS and etc) from impacting the overall availability of Consul cluster.

A northstar for this feature: Allow Consul operators to run large consul cluster(s) similar to a public Cloud Service like [Service Discovery](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html).

Here I want to keep the feature request at a very high level. There are many tactical steps and features that would be required for Consul to be fully multi-tenant ready. I do not want to prescribe the solutions here.

#### Ideas
- Many rate-limiting features are proposed in #12593. These would all help with multi-tenant isolation. Here we extend on a few more.
- Separate throttling for heavy-weight operations vs. light-weight operations. High RPS for heavy-weight operations can bring down consul clusters much more easily.
- Consider separate throttling for KV APIs vs. Service catalog APIs. Some large clusters might just disable KV API all together.
- Consider per-API RPS limit.

- Formalize the tenant concept in Consul. A tenant here can be a service owner, a Namespace, access Token etc.

- Formalize the tenant boundary and enforce it
- ACL for APIs?
- ACL for path?
- RPS limit per tenant?
- Quota system for tenants?
- Where to enforce the boundary? Ideally it should be enforced at the Consul server side instead of the agent side.

Contributor guide

Open the contributing guide

Research direction

Review the related rate-limiting proposals in #12593 and the issue's tenant candidates (service owner, Namespace, access token), then define the server-side boundary across KV and service-catalog APIs. Done requires a concrete tenant model and enforceable isolation, throttling, or quota behavior that prevents one tenant from affecting cluster availability.

Written by the indexing model from the issue text.

Assessment

Domain
backend-api-design, distributed-systems, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.