aws / aws/aws-ec2-instance-connect-config

eic_harvest_hostkeys fails in local zones

Open
#28 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
88
Forks
39
PR merge metrics
No merged PRs in 30d

Description

ec2-instance-connect breaks during host key harvesting for instances launched in [local zones](https://aws.amazon.com/about-aws/global-infrastructure/localzones/).

I have tested with Ubuntu 21.04 (Hirsute) using that distro's package (version 1.1.13-0ubuntu1) but inspection of the code here indicates the problem is also in this upstream code @ https://github.com/aws/aws-ec2-instance-connect-config/blob/c15b99fa223f277787e50b044baf39e483dedf8c/src/bin/eic_harvest_hostkeys#L101

### Recreate steps

1. Launch a VM in a local zone with ec2-instance-connect installed (I used Ubuntu in us-west-2-lax-1a).
2. Observe the state of the ec2-instance-connect systemd unit (it will be in the failed state)

```
$ systemctl is-system-running
degraded

$ systemctl list-units --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● ec2-instance-connect.service loaded failed failed EC2 Instance Connect Host Key Harvesting

$ journalctl --unit ec2-instance-connect
-- Logs begin at Wed 2021-02-10 22:47:47 UTC, end at Wed 2021-02-10 22:55:46 UTC. --
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Starting EC2 Instance Connect Host Key Harvesting...
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Main process exited, code=exited, status=255/EXCEPTION
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Failed with result 'exit-code'.
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Failed to start EC2 Instance Connect Host Key Harvesting.
```

3. Run the harvest hostkeys script with debug output via `bash -x /usr/share/ec2-instance-connect/eic_harvest_hostkeys` to observe the failure
```
...
++ /usr/bin/curl -s -f -m 1 -H 'X-aws-ec2-metadata-token:<>' http://169.254.169.254/latest/meta-data/placement/availability-zone/
+ zone=us-west-2-lax-1a
+ zone_exit=0
+ '[' 0 -ne 0 ']'
+ /bin/echo us-west-2-lax-1a
+ /bin/grep -Eq '^([a-z]+-){2,3}[0-9][a-z]$'
+ /usr/bin/head -n 1
+ exit 255
```

Contributor guide

Open the contributing guide

Research direction

Start with src/bin/eic_harvest_hostkeys, especially the availability-zone lookup and validation shown in the reproduction; compare it with the installed script and the referenced upstream revision. Reproduce the failure in an EC2 local zone and verify that the ec2-instance-connect systemd service completes host key harvesting without entering a failed state.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, shell
Domain
cloud, devops
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.