Multi-tenant Isolation For Consul Control Plane
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Context
We are running Consul in a multi-tenant fashion. We have large consul cluster(s) with many consul agents (>>10k). Many application teams registered a large number of services into Consul. Teams not only use Consul as a service catalog, some teams also use consul’s KV functionalities. As Consul usage grows organically, more load and variation of usage patterns grows significantly. Consul operators cannot effectively enforce best practices in reliability and scalability context for consul usage across all the teams. This impacts the reliability of our consul deployment significantly.
As our past incidents have shown, a single bad tenant (e.g. one service owned by one team) was able to bring down our consul cluster. Many services depend on Consul on their critical path. This in turn caused large scale outages across many services.
#### Feature Description
Support multi-tenant isolation natively in Consul. Facilitate running large consul deployments reliably in multi-tenant environment. Prevent one or few bad tenants (e.g. misbehaving, DDoS and etc) from impacting the overall availability of Consul cluster.
A northstar for this feature: Allow Consul operators to run large consul cluster(s) similar to a public Cloud Service like [Service Discovery](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html).
Here I want to keep the feature request at a very high level. There are many tactical steps and features that would be required for Consul to be fully multi-tenant ready. I do not want to prescribe the solutions here.
#### Ideas
- Many rate-limiting features are proposed in #12593. These would all help with multi-tenant isolation. Here we extend on a few more.
- Separate throttling for heavy-weight operations vs. light-weight operations. High RPS for heavy-weight operations can bring down consul clusters much more easily.
- Consider separate throttling for KV APIs vs. Service catalog APIs. Some large clusters might just disable KV API all together.
- Consider per-API RPS limit.
- Formalize the tenant concept in Consul. A tenant here can be a service owner, a Namespace, access Token etc.
- Formalize the tenant boundary and enforce it
- ACL for APIs?
- ACL for path?
- RPS limit per tenant?
- Quota system for tenants?
- Where to enforce the boundary? Ideally it should be enforced at the Consul server side instead of the agent side.
Contributor guide
Research direction
Review the related rate-limiting proposals in #12593 and the issue's tenant candidates (service owner, Namespace, access token), then define the server-side boundary across KV and service-catalog APIs. Done requires a concrete tenant model and enforceable isolation, throttling, or quota behavior that prevents one tenant from affecting cluster availability.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend-api-design, distributed-systems, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100