aws-samples / aws-samples/eks-workshop-v2

Karpenter node provisioning lab can fail under SCP due to c5.large in default NodePool

Open
#1,859 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
HCL
Stars
626
Forks
613
Avg merge
2d 7h
Merged PRs (30d)
9

Description

## Workshop page

https://www.eksworkshop.com/docs/fundamentals/compute/karpenter/node-provisioning

## What happened

While following the Karpenter node provisioning lab, the `inflate` pods stayed Pending and Karpenter did not provision nodes.

The cluster had a `NodePool/default` and `EC2NodeClass/default`, but the NodePool was not usable because the EC2NodeClass failed validation:

```text
EC2NodeClass default: Ready=False
ValidationSucceeded=False
Reason=RunInstancesAuthCheckFailed
Message: Controller isn't authorized to call ec2:RunInstances

NodePool default: Ready=False
Reason=UnhealthyDependents
Message: NodeClassReady=False
```

Karpenter controller logs showed:

```text
unauthorized to call ec2:RunInstances
... explicit deny in a service control policy ...
ignoring nodepool, not ready
no dynamic nodepools found
```

Decoding the authorization failure showed the SCP was denying invalid instance types. The account policy allowed patterns such as:

```text
t*.nano
t*.micro
t*.small
t*.medium
t*.large
t*.xlarge
m*.medium
m*.large
```

But the workshop NodePool includes:

```yaml
requirements:
- key: node.kubernetes.io/instance-type
operator: In
values:
- c5.large
- m5.large
- r5.large
- m5.xlarge
```

Karpenter attempted/validated `c5.large`, hit the SCP explicit deny, marked the EC2NodeClass not ready, and then ignored the whole NodePool instead of falling through to `m5.large`.

## Workaround

Restricting the NodePool to `m5.large` made the EC2NodeClass and NodePool ready, created a NodeClaim, launched an EC2 instance, and scheduled the pending pods:

```bash
kubectl patch nodepool default --type=json \
-p '[{"op":"replace","path":"/spec/template/spec/requirements/1/values","value":["m5.large"]}]'
```

After that:

```text
nodepool/default READY=True NODES=1
ec2nodeclass/default READY=True
nodeclaim/default-* TYPE=m5.large READY=True
inflate-* 1/1 Running
```

## Why this seems like a workshop issue

The lab assumes all listed instance types are allowed in the AWS environment. In environments with SCPs that constrain instance families/sizes, one denied candidate instance type can cause Karpenter's readiness validation to fail the entire NodePool, even when another listed type would be allowed.

A small note in the workshop, or using only broadly allowed workshop instance types, would make this easier to diagnose. For example, the page could mention checking `EC2NodeClass` conditions and adapting the NodePool instance-type list to match account/SCP restrictions.

## Environment observed

- EKS: 1.33
- Karpenter controller image/chart: 1.9.0
- Region: us-west-2

Contributor guide

Open the contributing guide

Research direction

Start with the linked Karpenter node-provisioning workshop page and its NodePool instance-type example. Check the documented lab's EC2NodeClass and NodePool conditions under an SCP-constrained account, then update the page with the relevant diagnosis and instance-type adjustment guidance. Done means the lab explains this failure mode and how to adapt the listed types.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.