aws / aws/amazon-vpc-resource-controller-k8s
Unpaginated calls to DescribeNetworkInterfaces getting blocked by EC2
- Dominant language
- Go
- Stars
- 106
- Forks
- 71
- Avg merge
- 10h 28m
- Merged PRs (30d)
- 6
Description
**Describe the Bug**:
We've been in the process of rolling out security groups for pods across various EKS clusters in multiple AWS accounts that we own. Recently, we've attempted to do the same in one of our "larger" AWS accounts, and noticed that pods would hang indefinitely waiting for an ENI to be attached by vpc-resource-controller.
We recently raised the ENI limits on this account to a very large value, and was told that unpaginated calls to `DescribeNetworkInterfaces` would be blocked as a result, and it seems that this is impacting vpc-resource-controller as you can see here in this CloudTrail event:
```json
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": ":amazon-vpc-resource-controller-k8s",
"arn": "arn:aws:sts:::assumed-role//amazon-vpc-resource-controller-k8s",
"accountId": "",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "",
"arn": "arn:aws:iam:::role/-control-20210716225641340500000009",
"accountId": "",
"userName": "-control-20210716225641340500000009"
},
"webIdFederationData": {},
"attributes": {
"creationDate": "2023-03-14T21:23:53Z",
"mfaAuthenticated": "false"
}
},
"invokedBy": "eks.amazonaws.com"
},
"eventTime": "2023-03-14T21:35:11Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "DescribeNetworkInterfaces",
"awsRegion": "us-east-1",
"sourceIPAddress": "eks.amazonaws.com",
"userAgent": "eks.amazonaws.com",
"errorCode": "Client.OperationNotPermitted",
"errorMessage": "This operation is not permitted.",
"requestParameters": {
"networkInterfaceIdSet": {},
"filterSet": {
"items": [
{
"name": "tag:vpcresources.k8s.aws/trunk-eni-id",
"valueSet": {
"items": [
{
"value": "eni-019dd4ee9f4aa06c8"
}
]
}
}
]
}
},
"responseElements": null,
"requestID": "8fd9723c-c4fa-4dd2-82be-47d8f328bf8e",
"eventID": "fd8c44a0-b04c-430d-9e77-8c478b0f2de3",
"readOnly": true,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "",
"eventCategory": "Management"
}
```
For AWS engineers, our case # is 12229078111
**Observed Behavior**:
Calls to `DescribeNetworkInterfaces` are blocked by EC2 with a `Client.OperationNotPermitted` error code
**Expected Behavior**:
SGP should work even if unpaginated calls are blocked.
**How to reproduce it (as minimally and precisely as possible)**:
Increase ENI limits in an AWS account high enough that EC2 blocks unpaginated calls to `DescribeNetworkInterfaces`
**Additional Context**:
**Environment**:
- Kubernetes version (use `kubectl version`): `Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.16-eks-ffeb93d", GitCommit:"52e500d139bdef42fbc4540c357f0565c7867a81", GitTreeState:"clean", BuildDate:"2022-11-29T18:41:42Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}`
- CNI Version 1.11.4
- OS (Linux/Windows): Amazon Linux (EKS AMI)
Contributor guide
Assessment
This issue has not been assessed yet.