aws-samples / aws-samples/eks-workshop-v2

Setup fails when VPC Block Public Access is enabled

Open
#1,928 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HCL
Stars
626
Forks
613
Avg merge
2d 7h
Merged PRs (30d)
9

Description

## Summary
The CloudFormation IDE [setup template](https://www.eksworkshop.com/docs/introduction/setup/your-account/) fails in AWS accounts where VPC Block Public Access is enabled in `block-ingress` mode.

The template creates a public EC2 instance with a public IP and an Internet Gateway route, then expects that instance to reach Systems Manager, Amazon Linux package repositories, GitHub, and COPR over HTTPS.

With VPC Block Public Access enabled, direct Internet Gateway traffic from the instance is blocked. The stack therefore fails during bootstrap.

## Environment
- Region: `us-west-2`
- [Template](https://ws-assets-prod-iad-r-pdx-f3b3f9f1a7d6a3d0.s3.us-west-2.amazonaws.com/39146514-f6d5-41cb-86ef-359f9d2f7265/eks-workshop-vscode-cfn.yaml
): `eks-workshop-vscode-cfn.yaml`
- Repository reference: `stable`
- AMI: Amazon Linux 2023
- Account configuration: VPC Block Public Access enabled with
`InternetGatewayBlockMode=block-ingress`

## Observed behavior
The setup failed in multiple stages:
1. The custom resource tried to call SSM before the EC2 instance registered
as a managed node:
```text
InvalidInstanceId: Instances not in a valid state for account
2. After SSM connectivity was added, the Amazon Linux package repository timed
out because it is S3-backed:
```text
Curl error (28): Timeout was reached for
al2023-repos-...s3.dualstack.us-west-2.amazonaws.com
3. After adding private S3 access, the bootstrap preflight showed that GitHub was still unreachable:
```text
Network preflight failed: cannot reach
raw.githubusercontent.com
**The original template assumes direct public Internet Gateway access works.**

## Expected behavior
The setup documentation or template should explicitly support accounts with VPC Block Public Access enabled or at least detect if that is the root cause of the stack creation failure.

## Suggested improvement
One of the following could be implemented:

1. Document VPC Block Public Access as a setup prerequisite and explain that direct Internet Gateway egress is required by the default IDE template.

2. Add a clearly opt-in template parameter for a scoped `AWS::EC2::VPCBlockPublicAccessExclusion` on the temporary workshop VPC. This should not be enabled by default because it deliberately creates a VPC exception to an account-level security control.

3. Improve bootstrap reliability by:
- Waiting for `SSM PingStatus=Online` before `SendCommand`
- Retrying transient `InvalidInstanceId` failures
- Adding private endpoints for `ssm`, `ssmmessages`, and `ec2messages`
- Adding an S3 Gateway endpoint for Amazon Linux package repositories
- Adding a fast connectivity preflight for S3, GitHub, and COPR before starting the long bootstrap script

## Local workaround used
I was able to complete setup by using a modified version of the workshop template that:

- Added a scoped VPC Block Public Access exclusion for only the temporary workshop VPC
- Added SSM interface endpoints and an S3 Gateway endpoint
- Waited for the instance to be SSM Online
- Added network preflight checks before installation

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked eks-workshop-vscode-cfn.yaml template and the setup documentation, then reproduce the stack failure with VPC Block Public Access enabled in block-ingress mode. The issue presents several possible fixes, so confirm the intended approach before changing anything; done means the chosen support or detection path works reliably and is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.