aws / aws/aws-ec2-instance-connect-config
Key trust verification fails on Ubuntu 22.04
- Dominant language
- Shell
- Stars
- 88
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
I can't get EC2 instance connect to work on Ubuntu 22.04 due to what seems to be an issue with trust chain verification.
To reproduce:
1. Launch an EC2 instance with the current Ubuntu 22.04 AMI (e.g. `ami-0aeb7c931a5a61206` in us-east-2).
2. Try to connect to it via `mssh ubuntu@`.
3. Observe that the command fails with "Permission denied (publickey)."
When using the `--debug` flag with `mssh`, I see that the public key is pushed successfully, but the remote rejects the connection:
```
...
2022-05-06 09:10:58,549 - EC2InstanceConnect - DEBUG - Successfully got instance information from EC2 API for
...
2022-05-06 09:10:59,189 - EC2InstanceConnect - DEBUG - Successfully pushed the public key to
2022-05-06 09:10:59,190 - EC2InstanceConnect - DEBUG - Generated command: ssh -o "IdentitiesOnly=yes" -i /var/folders/30/xdglsm2j3tz1rn1n7yygtm7c0000gn/T/tmp33a253uf ubuntu@
ubuntu@: Permission denied (publickey).
2022-05-06 09:10:59,612 - EC2InstanceConnect - DEBUG - Deleting the private key file: /var/folders/30/xdglsm2j3tz1rn1n7yygtm7c0000gn/T/tmp33a253uf
```
On the instance side, the following error is logged:
```
AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:wiFxouWj6qQ0aUZ0CAcftWZqNEf3qj2LLicCfGFcQJY failed, status 2
```
When I make these changes to `/usr/share/ec2-instance-connect/eic_parse_authorized_keys`:
```diff
@@ -17,8 +17,6 @@
# Openssl to use provided as $1
# Signer public key file path provided as $2
-set -e
-
# Set umask so only we can touch temp files
umask 077
@@ -165,7 +163,7 @@
# Verify the trust chain
if [ -d "${ca_path}" ] ; then
- verify_out=$("${OPENSSL}" verify -x509_strict -CApath "${ca_path}" -CAfile "${tmpdir}/ca-trust.pem" "${tmpdir}/cert.pem")
+ verify_out=$("${OPENSSL}" verify -x509_strict -CApath "${ca_path}" -CAfile "${tmpdir}/ca-trust.pem" "${tmpdir}/cert.pem" 2>&1)
verify_status=$?
else
# If the CA path is not a directory then do not use it - openssl will throw errors on versions 1.1.1+
@@ -173,7 +171,8 @@
verify_status=$?
fi
if [ $verify_status -ne 0 ] || [ "${verify_out}" != "${tmpdir}/cert.pem: OK" ] ; then
- fail "${is_debug}" "EC2 Instance Connect could not verify the signer trust chain. No keys have been trusted."
+ fail "${is_debug}" "EC2 Instance Connect could not verify the signer trust chain. No keys have been trusted.
+$verify_out"
fi
# Verify no certificates have been revoked
```
... the actual error is revealed:
```
EC2 Instance Connect could not verify the signer trust chain. No keys have been trusted.
C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
error 89 at 4 depth lookup: Basic Constraints of CA cert not marked critical
C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
error 92 at 4 depth lookup: CA cert does not include key usage extension
error /dev/shm/eic-l8wtfU3e/cert.pem: verification failed
```
It seem like the issue is either with `mssh` generating invalid keys or with `ec2-instance-connect` performing a trust verification that is too strict.
Contributor guide
Research direction
Start with /usr/share/ec2-instance-connect/eic_parse_authorized_keys and the eic_run_authorized_keys invocation shown in the instance log. Reproduce the failure on Ubuntu 22.04 with mssh --debug, then inspect the OpenSSL trust-chain verification output; done means the signer chain is handled correctly and the connection no longer ends with Permission denied (publickey).
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, shell
- Domain
- authentication, operating-systems, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100