aws-samples / aws-samples/eks-workshop-v2
Karpenter node provisioning lab can fail under SCP due to c5.large in default NodePool
- Dominant language
- HCL
- Stars
- 626
- Forks
- 613
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 9
Description
## Workshop page
https://www.eksworkshop.com/docs/fundamentals/compute/karpenter/node-provisioning
## What happened
While following the Karpenter node provisioning lab, the `inflate` pods stayed Pending and Karpenter did not provision nodes.
The cluster had a `NodePool/default` and `EC2NodeClass/default`, but the NodePool was not usable because the EC2NodeClass failed validation:
```text
EC2NodeClass default: Ready=False
ValidationSucceeded=False
Reason=RunInstancesAuthCheckFailed
Message: Controller isn't authorized to call ec2:RunInstances
NodePool default: Ready=False
Reason=UnhealthyDependents
Message: NodeClassReady=False
```
Karpenter controller logs showed:
```text
unauthorized to call ec2:RunInstances
... explicit deny in a service control policy ...
ignoring nodepool, not ready
no dynamic nodepools found
```
Decoding the authorization failure showed the SCP was denying invalid instance types. The account policy allowed patterns such as:
```text
t*.nano
t*.micro
t*.small
t*.medium
t*.large
t*.xlarge
m*.medium
m*.large
```
But the workshop NodePool includes:
```yaml
requirements:
- key: node.kubernetes.io/instance-type
operator: In
values:
- c5.large
- m5.large
- r5.large
- m5.xlarge
```
Karpenter attempted/validated `c5.large`, hit the SCP explicit deny, marked the EC2NodeClass not ready, and then ignored the whole NodePool instead of falling through to `m5.large`.
## Workaround
Restricting the NodePool to `m5.large` made the EC2NodeClass and NodePool ready, created a NodeClaim, launched an EC2 instance, and scheduled the pending pods:
```bash
kubectl patch nodepool default --type=json \
-p '[{"op":"replace","path":"/spec/template/spec/requirements/1/values","value":["m5.large"]}]'
```
After that:
```text
nodepool/default READY=True NODES=1
ec2nodeclass/default READY=True
nodeclaim/default-* TYPE=m5.large READY=True
inflate-* 1/1 Running
```
## Why this seems like a workshop issue
The lab assumes all listed instance types are allowed in the AWS environment. In environments with SCPs that constrain instance families/sizes, one denied candidate instance type can cause Karpenter's readiness validation to fail the entire NodePool, even when another listed type would be allowed.
A small note in the workshop, or using only broadly allowed workshop instance types, would make this easier to diagnose. For example, the page could mention checking `EC2NodeClass` conditions and adapting the NodePool instance-type list to match account/SCP restrictions.
## Environment observed
- EKS: 1.33
- Karpenter controller image/chart: 1.9.0
- Region: us-west-2
Contributor guide
Research direction
Start with the linked Karpenter node-provisioning workshop page and its NodePool instance-type example. Check the documented lab's EC2NodeClass and NodePool conditions under an SCP-constrained account, then update the page with the relevant diagnosis and instance-type adjustment guidance. Done means the lab explains this failure mode and how to adapt the listed types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100