aws / aws/amazon-vpc-cni-plugins

On a host with no IPv6 capabilities the plugin crashes when blocking the IMDS endpoint

Open
#106 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
68
Forks
54
PR merge metrics
No merged PRs in 30d

Description

### Summary
Since v1.69.0 of the ECS agent the plugin crashes, with the latest version, 1.77.0, the service can start a task, but it cannot connect to the network. As per AWS recommendations we're using awsvpc mode and blocking the IMDS endpoint, which appears to be the cause of the issue.

### Description
We see an error in the logs, see below, when the plugin attempts to add the blackhole route for the IPv6 address of the IMDS, this [line](https://github.com/aws/amazon-vpc-cni-plugins/blob/be5214353252f8315a1341f4df9ffbd8cf69000c/network/imds/imds_linux.go#L37) in the code appears to be what's failing.

![image](https://github.com/aws/amazon-vpc-cni-plugins/assets/5005972/31e75b4c-fc7d-4593-90d1-6e8e8fdd4dae)

The AMI that we use for the EC2 instance does not have IPv6 enabled for security reasons.

### Expected Behavior
The plugin handles the resulting error gracefully, or provides some means of disabling the IPv6 support to avoid the error.

### Observed Behavior
The plugin crashes and fails to finish configuring the network routes for the task, e.g. the default route is missing from the task route table, etc.

Route table with latest agent and `ECS_AWSVPC_BLOCK_IMDS=true`:
```
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
169.254.169.254 * 255.255.255.255 UH 0 0 0 *
169.254.170.2 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
169.254.172.1 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
```

Route table with latest agent and `ECS_AWSVPC_BLOCK_IMDS=false`:
```
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default ip-10-116-128-1 0.0.0.0 UG 0 0 0 eth0
10.116.128.128 * 255.255.255.224 U 0 0 0 eth0
169.254.170.2 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
169.254.172.1 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
```

Route table with v1.68.2 of the agent and `ECS_AWSVPC_BLOCK_IMDS=true`:
```
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default ip-10-116-128-1 0.0.0.0 UG 0 0 0 eth0
10.116.128.160 * 255.255.255.224 U 0 0 0 eth0
169.254.169.254 * 255.255.255.255 UH 0 0 0 *
169.254.170.2 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
169.254.172.1 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
```

### Environment Details
docker info:
```
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.0.0+unknown)

Server:
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 3
Server Version: 20.10.25
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
runc version: f19387a6bec4944c770f7668ab51c4348d9c2f38
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.197-186.748.amzn2.x86_64
Operating System: Amazon Linux 2
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.43GiB
Name: ip-X-X-X-X.eu-west-2.compute.internal
ID: U742:Y4OW:URO4:EH6C:LBN5:5HMC:HHOP:LWGQ:6RIV:FTJY:KJW3:QBBQ
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
```

Contributor guide

Open the contributing guide

Research direction

Start at network/imds/imds_linux.go, especially the route operation at line 37, and reproduce the failure with ECS_AWSVPC_BLOCK_IMDS=true on an EC2 host without IPv6 enabled. Done means the plugin handles the IPv6 route error without crashing and still finishes configuring the task's network routes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.