aws / aws/containers-roadmap

[ECS] ECS Agent Fails To Register Container Instance - AMZ-Linux-2 - IPv6-only

Open
#1,477 4 comments 6 reactions 1 assignee Claimed by @vibhav-ag View on GitHub
ECS
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Summary

The ecs-agent on my container instance can't register with my ECS service because it can't connect over IPv6. I believe this is because the ecs endpoint doesn't support IPv6.

### Description

I'm running a dual-stack setup in my private subnet, with private IPv4 addresses and public IPv6 addresses behind an egress-only internet gateway. When the ecs-agent starts, it attempts to register with the ECS service, but it can't connect so it eventually fails. Then my containers can never start.

On the other hand, if I put it in a public subnet it works fine. I assume this is because it can communicate with the ecs endpoint over a public IPv4 address.

### Expected Behavior
The ecs-agent should be able to register with an ECS service over an IPv6 connection.

### Observed Behavior
The ecs-agent cannot register with an ECS service over an IPv6 connection.

### Environment Details

Some things i can confirm from the box.

I'm running Amazon-Linux-2 (ECS-Optimized).
```
$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
```

The ecs agent installed and started correctly.
```
$ sudo systemctl status ecs
● ecs.service - Amazon Elastic Container Service - container agent
Loaded: loaded (/usr/lib/systemd/system/ecs.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2021-08-14 22:29:06 UTC; 3min 56s ago
Docs: https://aws.amazon.com/documentation/ecs/
Process: 4093 ExecStartPre=/usr/libexec/amazon-ecs-init pre-start (code=exited, status=0/SUCCESS)
Main PID: 4207 (amazon-ecs-init)
Tasks: 7
Memory: 73.0M
CGroup: /system.slice/ecs.service
└─4207 /usr/libexec/amazon-ecs-init start
```

### Supporting Log Snippets

Basically this just keeps happening over and over again.

```
$ less /var/log/ecs/ecs-init.log
...
2021-08-14T22:30:14Z [INFO] Starting Amazon Elastic Container Service Agent
2021-08-14T22:30:35Z [INFO] Agent exited with code 1
2021-08-14T22:30:35Z [WARN] ECS Agent failed to start, retrying in 4.235010051s
2021-08-14T22:30:39Z [INFO] Container name: /ecs-agent
2021-08-14T22:30:39Z [INFO] Removing existing agent container ID: daf1a3cf3a283c761fe038e08dbedcf7fbac7870e89a41c45120693b525ee13f
```

```
$ less /var/log/ecs/ecs-agent.log
...
level=error time=2021-08-14T22:33:21Z msg="Unable to register as a container instance with ECS: RequestError: send request failed\ncaused by: Post \"https://ecs.us-east-2.amazonaws.com/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" module=client.go
level=error time=2021-08-14T22:33:21Z msg="Error registering: RequestError: send request failed\ncaused by: Post \"https://ecs.us-east-2.amazonaws.com/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" module=agent.go
```

Similar problem with cloudformation cfn-signal.
```
$ less /var/log/cfn-init.log
2021-08-14 22:25:56,687 [DEBUG] Sleeping for 0.895379 seconds before retrying
2021-08-14 22:25:57,583 [DEBUG] Signaling resource ECSAutoScalingGroup in stack myStack with unique ID i-0d2b8edf184330cde and status SUCCESS
2021-08-14 22:26:57,584 [WARNING] Timeout of 60 seconds breached
```

I can ping google's ipv6 checkpoint from the instance.
```
$ ping6 ipv6.google.com
PING ipv6.google.com(ord37s18-in-x0e.1e100.net (2607:f8b0:4009:805::200e)) 56 data bytes
64 bytes from ord37s18-in-x0e.1e100.net (2607:f8b0:4009:805::200e): icmp_seq=1 ttl=96 time=18.3 ms
64 bytes from ord37s18-in-x0e.1e100.net (2607:f8b0:4009:805::200e): icmp_seq=2 ttl=96 time=18.4 ms
^C
--- ipv6.google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
```

But I cannot reach ecs.
```
$ ping6 https://ecs.us-east-2.amazonaws.com
ping: https://ecs.us-east-2.amazonaws.com: Name or service not known
```

Because they don't have a AAAA record.
```
$ dig AAAA https://ecs.us-east-2.amazonaws.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.5.2 <<>> AAAA https://ecs.us-east-2.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 59003
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
```

Nor does cloudformation.
```
$ dig AAAA https://cloudformation.us-east-2.amazonaws.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.5.2 <<>> AAAA https://cloudformation.us-east-2.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55794
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
```

😿

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.