Created subnets are unnecessarily small
- Dominant language
- Go
- Stars
- 16.7k
- Forks
- 4.7k
- Avg merge
- 17h 21m
- Merged PRs (30d)
- 68
Description
**1. What `kops` version are you running? The command `kops version`, will display
this information.**
1.12.1
**2. What Kubernetes version are you running? `kubectl version` will print the
version if a cluster is running or provide the Kubernetes version specified as
a `kops` flag.**
1.12.9
**3. What cloud provider are you using?**
AWS
**4. What commands did you run? What is the simplest way to reproduce this issue?**
```
kops create cluster --associate-public-ip=False --bastion=True --cloud=aws --dns-zone=k8s.domain.com --image 595879546273/CoreOS-stable-2135.4.0-hvm --master-size=t3.large --master-zones=eu-central-1a,eu-central-1b,eu-central-1c --master-count=3 --master-volume-size=100 --network-cidr=172.31.239.0/24 --networking=calico --node-count=1 --node-size=m5a.2xlarge --node-volume-size=300 --target=direct --topology=private --zones=eu-central-1a,eu-central-1b,eu-central-1c --ssh-public-key=~/.ssh/k8s-dev-id_rsa.pub --name=dev-qa.k8s.domain.com --state s3://my-state-store --authorization=rbac --encrypt-etcd-storage --kubernetes-version 1.12.9
```
`kops update cluster --name=dev-qa.k8s.domain.com --state s3://my-state-store --yes`
**5. What happened after the commands executed?**
KOPS fails to create resources due to "error creating subnet: InvalidSubnet.Range: The CIDR '172.31.239.8/30' is invalid."
Three normal /27 subnets were created (in the cluster config), Attempts were made to create (too small) /30 subnets for the utility subnets. This would use around half (/25) of the given IP range for the subnets, leaving the rest unused.
**6. What did you expect to happen?**
Three normal /26 subnets should be created and three /28 utility subnets should be created, using most of the given address space in the VPC. (One per AZ per type)
If the VPC is shared with other things, smaller subnets can be used.
**7. Please provide your cluster manifest. Execute
`kops get --name my.example.com -o yaml` to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.**
https://gist.github.com/mohag/2244eb229b15e776fd65b84a35a8cfb0
**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
Paste the logs into this report, or in a gist and provide the gist link here.**
Issue is visible in the generated manifest.
https://gist.github.com/mohag/2244eb229b15e776fd65b84a35a8cfb0#file-kops-create-cluster-log
**9. Anything else do we need to know?**
This seems to match the behaviour [described in the comments here](https://github.com/kubernetes/kops/blob/master/upup/pkg/fi/cloudup/subnets.go#L105). Ideally the number of subnets split into should be based on max('the number of availability zones specified',3)+1. (It allows for at least three AZs to be added. I'm not aware of providers with more than that in a region and if they do exist and are specified when the cluster is created, the subnet sizes can be reduced proportionally)
(Domain name and s3 bucket name substituted)
Manually substituting the subnets part with
```
subnets:
- cidr: 172.31.239.64/26
name: eu-central-1a
type: Private
zone: eu-central-1a
- cidr: 172.31.239.128/26
name: eu-central-1b
type: Private
zone: eu-central-1b
- cidr: 172.31.239.192/26
name: eu-central-1c
type: Private
zone: eu-central-1c
- cidr: 172.31.239.0/28
name: utility-eu-central-1a
type: Utility
zone: eu-central-1a
- cidr: 172.31.239.16/28
name: utility-eu-central-1b
type: Utility
zone: eu-central-1b
- cidr: 172.31.239.32/28
name: utility-eu-central-1c
type: Utility
zone: eu-central-1c
```
gets them created. The `kops update` then also succeed.
The created cluster validates as well.
Contributor guide
Research direction
Start in upup/pkg/fi/cloudup/subnets.go around the line referenced in the issue, then inspect the generated manifest and the linked log gist to understand the current CIDR allocation. Done means the requested availability zones receive appropriately sized normal and utility subnets, and the sample kops update and cluster validation succeed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, infrastructure, networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100