kubernetes / kubernetes/cloud-provider-aws

Allow specifying list of subnet IDs (rather than just a singular one)

Open
#1,201 9 comments 0 reactions 0 assignees View on GitHub
kind/feature needs-triage
Dominant language
Go
Stars
465
Forks
393
Avg merge
22h 25m
Merged PRs (30d)
11

Description

**What would you like to be added**:

I would like for an alternate option [to this one in the config](https://github.com/kubernetes/cloud-provider-aws/blob/cd72edc31806dd4e99fa84b2a8dcb708d428ee24/pkg/providers/v1/config/config.go#L48-L49) that allows me to specify multiple subnets to use, rather than just one. If necessary, this could be divided up into sub-lists for the private vs public subnets, although it feels like there could be a middle ground in which tags are used to distinguish the difference between those from the configuration's list.

**Why is this needed**:

We have two kinds of problems.

1. Non-EKS and EKS crossing wires:
1. We have multiple sets of private subnets, but the amount is static because IPAM is hard.
2. Because we need to support variable numbers of clusters per environment (which can shrink and grow,) we reuse the subnets.
3. In order to support this, we intentionally do not tag the subnets with `kubernetes.io/cluster/...`, since there are N cluster names per environment, and we'd hit our limits on them quickly; also, defining the tags with [`ec2_tag`](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_tag) is not compatible with the existing terraform definitions for the subnets.
4. We have found that non-EKS clusters will select subnets meant for the EKS clusters, due to lexicographical ordering. We believe that the reverse would be true, but the IDs just happen to all be such that the order shakes out that way across our environments. Either way, it's troublesome.
2. We're not able to ignore an AZ we don't want to use:
1. When we got started with AWS, we used all `us-east-1` AZs available to us. However, we discovered over time that not all AZs in that DC are created equal; when nitro instances were released, one of the AZs lacked support for them for a very long time (I think it was over a year.)
2. We decided that it would be risky to try to remove it from our network entirely, but we reorganized our then-fledgling Kubernetes terraform code to ignore the AZ that lacked nitro support. I think there were other problems besides nitro instance support, so we decided it should just forever be excluded. It's got subnets in the VPC but we don't have any nodes created in them.
3. Nonetheless, the cloud-controller-manager selects the undesirable AZ's subnets when making LBs, causing extra unnecessary network hops.
4. Since #442 was not implemented as described, we can't make the cloud-controller-manager ignore this AZ's subnets. We've considered putting in a tag with a fake cluster name to make [this code](https://github.com/kubernetes/cloud-provider-aws/blob/8ed070cd7501f6032ac056a4e79b9afc70fe522c/pkg/providers/v1/aws.go#L3512-L3519) not choose the subnets when auto-discovering, but that seems excessively janky and reliant upon an undocumented implementation detail that could break at any time.

Configuring subnets via annotation (as suggested [here](https://github.com/kubernetes/cloud-provider-aws/issues/249#issuecomment-906595623)) is an unwieldy fallback. Looking up the subnet IDs in advance (or using some templating tool to do it) is clunkier than just setting it at the provider level, and mutating webhooks have proven to be unreliable and problematic for us before.

Also, we're not sure if there's some other yet-undiscovered time bomb because of the subnet auto-discovery that applies to something besides LoadBalancer syncing, so it feels like something that should just be solved at the source. Being able to specify a list of subnets, rather than just one, feels like common sense.

/kind feature

Contributor guide

Open the contributing guide

Research direction

Start with the provider configuration in pkg/providers/v1/config/config.go at the linked subnet settings, then trace subnet auto-discovery in pkg/providers/v1/aws.go around lines 3512-3519. Clarify how a configured list should interact with private/public subnets and discovery; done means multiple subnet IDs can be configured and unwanted AZ subnets are not selected.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.