aws / aws/amazon-vpc-resource-controller-k8s

Pods requesting vpc.amazonaws.com/pod-eni cannot be scheduled - allocatable not set

Open
#637 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
106
Forks
71
Avg merge
10h 28m
Merged PRs (30d)
6

Description

**Describe the Bug**:
The VPC resource controller advertises extended resource capacity for branch ENIs (`vpc.amazonaws.com/pod-eni`) by patching `node.Status.Capacity`. However, `node.Status.Allocatable` remains unset, preventing the Kubernetes scheduler from scheduling pods that request these extended resources.

**Observed Behavior**:
When the controller advertises branch ENI capacity as an extended resource:
- `node.Status.Capacity["vpc.amazonaws.com/pod-eni"]` is set to the correct value by the controller
- `node.Status.Allocatable["vpc.amazonaws.com/pod-eni"]` remains unset
- Pods requesting `vpc.amazonaws.com/pod-eni` cannot be scheduled because the scheduler only considers allocatable resources

**Root Cause Analysis**:
The VPC Resource Controller only patches `Capacity`, not `Allocatable`. This can be verified in the code:
- `pkg/k8s/wrapper.go:185` - `AdvertiseCapacityIfNotSet` only sets `node.Status.Capacity`
- `pkg/provider/branch/provider.go:267` - Branch provider calls this function

According to Kubernetes design, the kubelet automatically calculates `Allocatable` from `Capacity` during periodic status sync (every ~10 seconds by default). For extended resources, `allocatable = capacity` since no system reservations apply.

However, there appears to be a timing issue where:
1. The controller patches Capacity after node initialization
2. The kubelet's periodic sync may not properly populate Allocatable from the externally-patched Capacity
3. This could be due to strategic merge patch behavior or the order of operations in kubelet's node status update logic
4. Our specific kubelet configuration may be causing this issue

**Expected Behavior**:
The scheduler should be able to schedule pods requesting `vpc.amazonaws.com/pod-eni` resources. For this to work, `node.Status.Allocatable["vpc.amazonaws.com/pod-eni"]` must be set to a non-zero value matching the capacity.

**How to reproduce it (as minimally and precisely as possible)**:
1. Enable Security Groups for Pods (`ENABLE_POD_ENI=true`)
2. Wait for the controller to advertise branch ENI capacity
3. Check node status: `kubectl get node -o json | jq '.status.capacity, .status.allocatable'`
4. Observe that capacity is set but allocatable is not
5. Create a pod requesting `vpc.amazonaws.com/pod-eni`
6. Pod remains in Pending state with event: "0/N nodes are available: N Insufficient vpc.amazonaws.com/pod-eni"

**Workaround**:
Manually patch allocatable:
```bash
kubectl patch node --subresource=status --type=merge -p '{"status":{"allocatable":{"vpc.amazonaws.com/pod-eni":"9"}}}'
```

**Additional Context**:
While the kubelet is designed to automatically calculate allocatable from capacity, this is not happening reliably in our stack.

**Environment**:
- Kubernetes version: 1.30+
- VPC Resource Controller Version: 1.7.x
- OS: Custom Ubuntu AMI on EKS

Contributor guide

Open the contributing guide

Research direction

Start by reading pkg/k8s/wrapper.go:185 and pkg/provider/branch/provider.go:267, then reproduce the node status and pending-pod behavior with the kubectl and jq steps in the issue. Trace why the advertised extended resource is missing from Status.Allocatable and verify that the scheduler can place a pod requesting vpc.amazonaws.com/pod-eni once the issue is fixed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.