aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap
AWS::EC2::SecurityGroup - PFR - provide error + information about duplicate Ingress/Egress rules via CFN stack-events
- Dominant language
- No language data
- Stars
- 1.1k
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
### Name of the resource
AWS::EC2::SecurityGroup
### Resource name
_No response_
### Description
#### Starting point:
This issue is a follow up (aka. mitigation) of #612 as an PFR to request an extension of CloudFormation Stack-Events while creating or updating an `AWS::EC2::SecurityGroup` **AND** having a duplicated `SecurityGroupIngress` (or Egress) defintion in your CloudFormation template. We try to explain and balance the scope of the PFR.
Before we start, let's sum up the details and current behavior of CloudFormation. Please also check the part "Addtional backgroup for consideration" to understand the current impact of AWS users run into this issue.
We create a new stack. In a real-life scenario it could be an update of an existing stack and -for sure- there are often much more Ingress-Defintion in such templates.
```yaml
AWSTemplateFormatVersion: 2010-09-09
Parameters:
VpcId:
Type: String
Description: 'Enter the VPC ID where the Security Groups will be created'
Resources:
TestSg:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupDescription: Test duplicate ingress
VpcId: !Ref VpcId
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 8080
ToPort: 8080
CidrIp: 172.24.186.0/23
Description: EntryA
- IpProtocol: tcp
FromPort: 8080
ToPort: 8080
CidrIp: 172.24.186.0/23
Description: EntryB (Duplicate) # = it's the same like EntryA
- IpProtocol: tcp
FromPort: 9090
ToPort: 9090
CidrIp: 192.168.0.1/32
Description: EntryX
```
#### Results:
* CloudFormation stack-event just reporting "created successful". CloudFormation appears to accept it silently:

* SecurityGroup is created. A SecurityGroup doesn't allow duplicates (hence here EntryA is missing in the resource).

#### Issues:
* Even you defined (accidentally) three Ingress rules in CFN template, there are only two [= EntryA is missing]. Let's check CloudTail for more information - There are three API-requests from CloudFormation:

* While checking the Events, we see that "EntryA" was rejected during AuthorizeSecurityGroupIngress API-call with a return of **Client.InvalidPermission.Duplicate**:
```json
{
"eventVersion": "1.08",
"userIdentity": {
"invokedBy": "cloudformation.amazonaws.com"
},
"eventTime": "2023-02-19T09:25:00Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "AuthorizeSecurityGroupIngress",
"awsRegion": "eu-central-1",
"sourceIPAddress": "cloudformation.amazonaws.com",
"userAgent": "AWS Internal",
"errorCode": "Client.InvalidPermission.Duplicate",
"errorMessage": "the specified rule \"peer: 172.24.186.0/23, TCP, from port: 8080, to port: 8080, ALLOW\" already exists",
"requestParameters": {
"groupId": "sg-0d08xxxxxxxxxxxxx",
"ipPermissions": {
"items": [
{
"ipProtocol": "tcp",
"fromPort": 8080,
"toPort": 8080,
"groups": {},
"ipRanges": {
"items": [
{
"cidrIp": "172.24.186.0/23",
"description": "EntryA"
}
]
},
"ipv6Ranges": {},
"prefixListIds": {}
}
]
}
},
"responseElements": null,
"requestID": "123-same-request-Id789",
"eventID": "123-same-event-Id789",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "xxxxxxxxxxxxx",
"eventCategory": "Management",
"sessionCredentialFromConsole": "true"
}
```
### Scope of the PFR
- [ ] CloudFormation _must_ provide the information about duplicate Ingress/Egress rule as an CloudFormation stack-event, e.g. in column "Status reson". So User getting an INFORMATION what is "wrong" in your IaC/CloudFormation template. Below a mockup of such output (should cover Stack-Create + Stack-Update):
| Timestamp | Logical ID | Status | Status reson |
| :--------- | :--------- | :--------- | :--------- |
| 2023-02-19 10:25:03 UTC+0100 | rogo-secgroup-stack | CREATE_COMPLETE | - |
| 2023-02-19 10:25:00 UTC+0100 | TestSg | CREATE_COMPLETE | **Received response status from AuthorizeSecurityGroupIngress with Client.InvalidPermission.Duplicate: "the specified rule \"peer: 172.24.186.0/23, TCP, from port: 8080, to port: 8080, ALLOW\" already exists". You might defined a duplicate rule. (RequestId: 123-same-request-Id789)** |
| 2023-02-19 10:24:59 UTC+0100 | TestSg | CREATE_IN_PROGRESS | Resource creation Initiated |
| 2023-02-19 10:24:54 UTC+0100 | TestSg | CREATE_IN_PROGRESS | - |
| 2023-02-19 10:24:51 UTC+0100 | rogo-secgroup-stack | CREATE_IN_PROGRESS | User Initiated |
- [ ] CloudFormation _could_ validate the template before send the three API-calls. It's debatable if this in an template validation error to define such duplicate Ingress/Egress rule in YAML or JSON. We are aware that such change is not expectable, since it would change the current behaviour for other AWS users.
- [ ] CloudFormation _should_ inform that just removing this duplicate entry from template and update again, could cause bigger issues - see details below.
### Addtional backgroup for consideration
_You might ask yourself, why we need this?_
Stacks with SecurityGroups are an ongoing config-set of rules, which are extended based on different requests. It is very likely that at some point somebody requesting to add an Ingress, even this IP/Port is already part of the template.
If you (or your coworker one year later with with the best of intentions) remove duplicated SecurityGroupIngress/Egress rule, you will run into the details of #612. So, let's follow up here based on the example We check the SecurityGroup resource and see that "EntryA" is not part of the SecurityGroup - Hence we remove this config from CloudFormation template:
```yaml
AWSTemplateFormatVersion: 2010-09-09
Parameters:
VpcId:
Type: String
Description: 'Enter the VPC ID where the Security Groups will be created'
Resources:
TestSg:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupDescription: Test duplicate ingress
VpcId: !Ref VpcId
SecurityGroupIngress:
# -->
# we removed EntryA since this rule is not part of the SecGroup
# <--
- IpProtocol: tcp
FromPort: 8080
ToPort: 8080
CidrIp: 172.24.186.0/23
Description: EntryB (Duplicate) # = it's the same Ingress-definition like EntryA
# keep EntryB, since it's part of the SecGroup as shown in Console
- IpProtocol: tcp
FromPort: 9090
ToPort: 9090
CidrIp: 192.168.0.1/32
Description: EntryX
```
We update the existing stack with this new template. In case we now checking the SecurityGroup again (since we doesn't expect any changes we might not even do this check), we see that also "EntryB" is not part of the SecurityGroup anymore! - Hence, we lost the connectivity for 172.24.186.0/23-port8080, which could cause a huge impact to business. Furthermore, our CloudFormation/IaC still show us this rule for 172.24.186.0/23-port8080. Here the Ingress rules of our SecurityGroup:

It seems that just the combination of Protocol/Port/Ip (as a diff on CloudFormation template level) identifies the rule to delete:

### Other Details
Once you are not aware of all this bloody details above, it's going to be a long night of debugging! ;-) 🥂
Contributor guide
Research direction
The issue provides CloudFormation YAML examples for AWS::EC2::SecurityGroup and describes duplicate ingress and egress behavior in stack events. Start by reviewing the duplicate-rule scenario and the context of #612; done would require an agreed CloudFormation behavior for reporting or validating these duplicates, but no repository files or tests are identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, yaml
- Domain
- cloud, infrastructure, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100