aws / aws/amazon-vpc-resource-controller-k8s

Unpaginated calls to DescribeNetworkInterfaces getting blocked by EC2

Open
#188 3 comments 0 reactions 1 assignee Assigned to @yash97 View on GitHub
bug
Dominant language
Go
Stars
106
Forks
71
Avg merge
10h 28m
Merged PRs (30d)
6

Description

**Describe the Bug**:

We've been in the process of rolling out security groups for pods across various EKS clusters in multiple AWS accounts that we own. Recently, we've attempted to do the same in one of our "larger" AWS accounts, and noticed that pods would hang indefinitely waiting for an ENI to be attached by vpc-resource-controller.

We recently raised the ENI limits on this account to a very large value, and was told that unpaginated calls to `DescribeNetworkInterfaces` would be blocked as a result, and it seems that this is impacting vpc-resource-controller as you can see here in this CloudTrail event:

```json
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": ":amazon-vpc-resource-controller-k8s",
"arn": "arn:aws:sts:::assumed-role//amazon-vpc-resource-controller-k8s",
"accountId": "",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "",
"arn": "arn:aws:iam:::role/-control-20210716225641340500000009",
"accountId": "",
"userName": "-control-20210716225641340500000009"
},
"webIdFederationData": {},
"attributes": {
"creationDate": "2023-03-14T21:23:53Z",
"mfaAuthenticated": "false"
}
},
"invokedBy": "eks.amazonaws.com"
},
"eventTime": "2023-03-14T21:35:11Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "DescribeNetworkInterfaces",
"awsRegion": "us-east-1",
"sourceIPAddress": "eks.amazonaws.com",
"userAgent": "eks.amazonaws.com",
"errorCode": "Client.OperationNotPermitted",
"errorMessage": "This operation is not permitted.",
"requestParameters": {
"networkInterfaceIdSet": {},
"filterSet": {
"items": [
{
"name": "tag:vpcresources.k8s.aws/trunk-eni-id",
"valueSet": {
"items": [
{
"value": "eni-019dd4ee9f4aa06c8"
}
]
}
}
]
}
},
"responseElements": null,
"requestID": "8fd9723c-c4fa-4dd2-82be-47d8f328bf8e",
"eventID": "fd8c44a0-b04c-430d-9e77-8c478b0f2de3",
"readOnly": true,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "",
"eventCategory": "Management"
}
```

For AWS engineers, our case # is 12229078111

**Observed Behavior**:

Calls to `DescribeNetworkInterfaces` are blocked by EC2 with a `Client.OperationNotPermitted` error code

**Expected Behavior**:

SGP should work even if unpaginated calls are blocked.

**How to reproduce it (as minimally and precisely as possible)**:

Increase ENI limits in an AWS account high enough that EC2 blocks unpaginated calls to `DescribeNetworkInterfaces`

**Additional Context**:

**Environment**:
- Kubernetes version (use `kubectl version`): `Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.16-eks-ffeb93d", GitCommit:"52e500d139bdef42fbc4540c357f0565c7867a81", GitTreeState:"clean", BuildDate:"2022-11-29T18:41:42Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}`
- CNI Version 1.11.4
- OS (Linux/Windows): Amazon Linux (EKS AMI)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.