aws-samples / aws-samples/eks-workshop-v2
Setup fails when VPC Block Public Access is enabled
- Dominant language
- HCL
- Stars
- 626
- Forks
- 613
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 9
Description
## Summary
The CloudFormation IDE [setup template](https://www.eksworkshop.com/docs/introduction/setup/your-account/) fails in AWS accounts where VPC Block Public Access is enabled in `block-ingress` mode.
The template creates a public EC2 instance with a public IP and an Internet Gateway route, then expects that instance to reach Systems Manager, Amazon Linux package repositories, GitHub, and COPR over HTTPS.
With VPC Block Public Access enabled, direct Internet Gateway traffic from the instance is blocked. The stack therefore fails during bootstrap.
## Environment
- Region: `us-west-2`
- [Template](https://ws-assets-prod-iad-r-pdx-f3b3f9f1a7d6a3d0.s3.us-west-2.amazonaws.com/39146514-f6d5-41cb-86ef-359f9d2f7265/eks-workshop-vscode-cfn.yaml
): `eks-workshop-vscode-cfn.yaml`
- Repository reference: `stable`
- AMI: Amazon Linux 2023
- Account configuration: VPC Block Public Access enabled with
`InternetGatewayBlockMode=block-ingress`
## Observed behavior
The setup failed in multiple stages:
1. The custom resource tried to call SSM before the EC2 instance registered
as a managed node:
```text
InvalidInstanceId: Instances not in a valid state for account
2. After SSM connectivity was added, the Amazon Linux package repository timed
out because it is S3-backed:
```text
Curl error (28): Timeout was reached for
al2023-repos-...s3.dualstack.us-west-2.amazonaws.com
3. After adding private S3 access, the bootstrap preflight showed that GitHub was still unreachable:
```text
Network preflight failed: cannot reach
raw.githubusercontent.com
**The original template assumes direct public Internet Gateway access works.**
## Expected behavior
The setup documentation or template should explicitly support accounts with VPC Block Public Access enabled or at least detect if that is the root cause of the stack creation failure.
## Suggested improvement
One of the following could be implemented:
1. Document VPC Block Public Access as a setup prerequisite and explain that direct Internet Gateway egress is required by the default IDE template.
2. Add a clearly opt-in template parameter for a scoped `AWS::EC2::VPCBlockPublicAccessExclusion` on the temporary workshop VPC. This should not be enabled by default because it deliberately creates a VPC exception to an account-level security control.
3. Improve bootstrap reliability by:
- Waiting for `SSM PingStatus=Online` before `SendCommand`
- Retrying transient `InvalidInstanceId` failures
- Adding private endpoints for `ssm`, `ssmmessages`, and `ec2messages`
- Adding an S3 Gateway endpoint for Amazon Linux package repositories
- Adding a fast connectivity preflight for S3, GitHub, and COPR before starting the long bootstrap script
## Local workaround used
I was able to complete setup by using a modified version of the workshop template that:
- Added a scoped VPC Block Public Access exclusion for only the temporary workshop VPC
- Added SSM interface endpoints and an S3 Gateway endpoint
- Waited for the instance to be SSM Online
- Added network preflight checks before installation
Contributor guide
Research direction
Start by reading the linked eks-workshop-vscode-cfn.yaml template and the setup documentation, then reproduce the stack failure with VPC Block Public Access enabled in block-ingress mode. The issue presents several possible fixes, so confirm the intended approach before changing anything; done means the chosen support or detection path works reliably and is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100