aws / aws/eks-anywhere

Bare metal ssh key changes on reboot

Open
#2,682 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
When I SSH into a system I provisioned on bare metal I can see the SSH key fingerprint
```
ssh ec2-user@10.10.1.53
The authenticity of host '10.10.1.53 (10.10.1.53)' can't be established.
ECDSA key fingerprint is SHA256:bA3B01cj2lzzmmGFBETC7rQcok7gYgIzMtWXqPpVpTI.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
```
When I reboot the system (not reprovisioning) the SSH key changes and I get a warning showing a new key fingerprint
```
ssh ec2-user@10.10.1.53
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:eyGntJB+/F3lHeHQfZWGVFVVoTI9ADrgCjB01K+kevo.
Please contact your system administrator.
```

**What you expected to happen**:
The SSH key should stay the same when rebooting and only change when re-provisioning the OS.

**How to reproduce it (as minimally and precisely as possible)**:
* Provision a host
* SSH into it (accept the key)
* reboot the host
* SSH into it again

**Anything else we need to know?**:
Listing the keys on the system before and after reboot didn't appear to be different but I suspect something in cloud-init is causing a new key to be created.

First boot
```
ssh-keyscan localhost
# localhost:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
# localhost:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
localhost ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLZttqagIsW8Sy8mvuQG9UE5i4Hsn5F928PYMOCmNcGTEidp/iuqLgodQXFGPdL47b4LAYMLbcYPD
r70KvTcgCKbPOtl2QqZB4KsdymFpmQA5szQOF78J5GjAEBZGL3RE05IWm9ni5VBJFLDjknE0yDipgNylWVWJi1CTtLXJNMqFR6uChQ5RQghFxpdBc9klwZiWm4FbH
JaysxjX4LGeWi6WR5z50PMIxGAhNSrS7qQlQ0UeimFWqUdBpyChrO2Jwe6yhl91Pzx4z20yGESaQYXZ5V6AK6MKtotvrWvKQAtFPoyQr991+vAh91xienepjMWgCF
kUKmWFxtFcqFRxoHmr7n/tgayfNcCBzABSmAORvCJ/07ixY2QNBVf7CY8tBnashXby5otgVPJ0aR1XyJQiAZk7ZvXDGddWXrSmt7HIYueGApu6z5IgmVlT3zir1iR
ycanbAyHHbwTI+KpkuZzQGiPlEyOlX4JXUGnY97H9P/NERFnDff/XJTwH4d+U=
# localhost:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
localhost ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD0lXXSkH4UsvsZMvqQrfH5Er2+TT6IRe0aCgVCPRt8
F/pneuRpjDVT1m3tt4nkASq7SU2T2OED5g2tkXYM6xgE=
# localhost:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
localhost ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO8ZbQ1TKjknr9okCvzu6Y/1br5RiqiRWFhsPri6qzN6
# localhost:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
```
after reboot
```
$ ssh-keyscan localhost
# localhost:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
# localhost:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
localhost ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLZttqagIsW8Sy8mvuQG9UE5i4Hsn5F928PYMOCmNcGTEidp/iuqLgodQXFGPdL47b4LAYMLbcYPD
r70KvTcgCKbPOtl2QqZB4KsdymFpmQA5szQOF78J5GjAEBZGL3RE05IWm9ni5VBJFLDjknE0yDipgNylWVWJi1CTtLXJNMqFR6uChQ5RQghFxpdBc9klwZiWm4FbH
JaysxjX4LGeWi6WR5z50PMIxGAhNSrS7qQlQ0UeimFWqUdBpyChrO2Jwe6yhl91Pzx4z20yGESaQYXZ5V6AK6MKtotvrWvKQAtFPoyQr991+vAh91xienepjMWgCF
kUKmWFxtFcqFRxoHmr7n/tgayfNcCBzABSmAORvCJ/07ixY2QNBVf7CY8tBnashXby5otgVPJ0aR1XyJQiAZk7ZvXDGddWXrSmt7HIYueGApu6z5IgmVlT3zir1iR
ycanbAyHHbwTI+KpkuZzQGiPlEyOlX4JXUGnY97H9P/NERFnDff/XJTwH4d+U=
# localhost:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
localhost ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD0lXXSkH4UsvsZMvqQrfH5Er2+TT6IRe0aCgVCPRt8
F/pneuRpjDVT1m3tt4nkASq7SU2T2OED5g2tkXYM6xgE=
# localhost:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
localhost ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO8ZbQ1TKjknr9okCvzu6Y/1br5RiqiRWFhsPri6qzN6
# localhost:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
```
**Environment**:
- EKS Anywhere Release: 0.10.0

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the bare-metal provisioning, SSH connection, and reboot sequence described in the issue, then inspect the cloud-init host-key handling suspected by the reporter. Compare the SSH host keys before and after reboot with ssh-keyscan; done means the keys remain stable across reboot and change only after reprovisioning.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, ubuntu
Domain
infrastructure, operating-systems, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.