aws / aws/aws-ec2-instance-connect-config
eic_harvest_hostkeys fails in local zones
- Dominant language
- Shell
- Stars
- 88
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
ec2-instance-connect breaks during host key harvesting for instances launched in [local zones](https://aws.amazon.com/about-aws/global-infrastructure/localzones/).
I have tested with Ubuntu 21.04 (Hirsute) using that distro's package (version 1.1.13-0ubuntu1) but inspection of the code here indicates the problem is also in this upstream code @ https://github.com/aws/aws-ec2-instance-connect-config/blob/c15b99fa223f277787e50b044baf39e483dedf8c/src/bin/eic_harvest_hostkeys#L101
### Recreate steps
1. Launch a VM in a local zone with ec2-instance-connect installed (I used Ubuntu in us-west-2-lax-1a).
2. Observe the state of the ec2-instance-connect systemd unit (it will be in the failed state)
```
$ systemctl is-system-running
degraded
$ systemctl list-units --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● ec2-instance-connect.service loaded failed failed EC2 Instance Connect Host Key Harvesting
$ journalctl --unit ec2-instance-connect
-- Logs begin at Wed 2021-02-10 22:47:47 UTC, end at Wed 2021-02-10 22:55:46 UTC. --
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Starting EC2 Instance Connect Host Key Harvesting...
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Main process exited, code=exited, status=255/EXCEPTION
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Failed with result 'exit-code'.
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Failed to start EC2 Instance Connect Host Key Harvesting.
```
3. Run the harvest hostkeys script with debug output via `bash -x /usr/share/ec2-instance-connect/eic_harvest_hostkeys` to observe the failure
```
...
++ /usr/bin/curl -s -f -m 1 -H 'X-aws-ec2-metadata-token:<>' http://169.254.169.254/latest/meta-data/placement/availability-zone/
+ zone=us-west-2-lax-1a
+ zone_exit=0
+ '[' 0 -ne 0 ']'
+ /bin/echo us-west-2-lax-1a
+ /bin/grep -Eq '^([a-z]+-){2,3}[0-9][a-z]$'
+ /usr/bin/head -n 1
+ exit 255
```
Contributor guide
Research direction
Start with src/bin/eic_harvest_hostkeys, especially the availability-zone lookup and validation shown in the reproduction; compare it with the installed script and the referenced upstream revision. Reproduce the failure in an EC2 local zone and verify that the ec2-instance-connect systemd service completes host key harvesting without entering a failed state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, shell
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100