aws / aws/aws-ec2-instance-connect-config

Key trust verification fails on Ubuntu 22.04

Open
#38 6 comments 2 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
88
Forks
39
PR merge metrics
No merged PRs in 30d

Description

I can't get EC2 instance connect to work on Ubuntu 22.04 due to what seems to be an issue with trust chain verification.

To reproduce:

1. Launch an EC2 instance with the current Ubuntu 22.04 AMI (e.g. `ami-0aeb7c931a5a61206` in us-east-2).
2. Try to connect to it via `mssh ubuntu@`.
3. Observe that the command fails with "Permission denied (publickey)."

When using the `--debug` flag with `mssh`, I see that the public key is pushed successfully, but the remote rejects the connection:

```
...
2022-05-06 09:10:58,549 - EC2InstanceConnect - DEBUG - Successfully got instance information from EC2 API for
...
2022-05-06 09:10:59,189 - EC2InstanceConnect - DEBUG - Successfully pushed the public key to
2022-05-06 09:10:59,190 - EC2InstanceConnect - DEBUG - Generated command: ssh -o "IdentitiesOnly=yes" -i /var/folders/30/xdglsm2j3tz1rn1n7yygtm7c0000gn/T/tmp33a253uf ubuntu@
ubuntu@: Permission denied (publickey).
2022-05-06 09:10:59,612 - EC2InstanceConnect - DEBUG - Deleting the private key file: /var/folders/30/xdglsm2j3tz1rn1n7yygtm7c0000gn/T/tmp33a253uf
```

On the instance side, the following error is logged:

```
AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:wiFxouWj6qQ0aUZ0CAcftWZqNEf3qj2LLicCfGFcQJY failed, status 2
```

When I make these changes to `/usr/share/ec2-instance-connect/eic_parse_authorized_keys`:

```diff
@@ -17,8 +17,6 @@
# Openssl to use provided as $1
# Signer public key file path provided as $2

-set -e
-
# Set umask so only we can touch temp files
umask 077

@@ -165,7 +163,7 @@

# Verify the trust chain
if [ -d "${ca_path}" ] ; then
- verify_out=$("${OPENSSL}" verify -x509_strict -CApath "${ca_path}" -CAfile "${tmpdir}/ca-trust.pem" "${tmpdir}/cert.pem")
+ verify_out=$("${OPENSSL}" verify -x509_strict -CApath "${ca_path}" -CAfile "${tmpdir}/ca-trust.pem" "${tmpdir}/cert.pem" 2>&1)
verify_status=$?
else
# If the CA path is not a directory then do not use it - openssl will throw errors on versions 1.1.1+
@@ -173,7 +171,8 @@
verify_status=$?
fi
if [ $verify_status -ne 0 ] || [ "${verify_out}" != "${tmpdir}/cert.pem: OK" ] ; then
- fail "${is_debug}" "EC2 Instance Connect could not verify the signer trust chain. No keys have been trusted."
+ fail "${is_debug}" "EC2 Instance Connect could not verify the signer trust chain. No keys have been trusted.
+$verify_out"
fi

# Verify no certificates have been revoked
```

... the actual error is revealed:

```
EC2 Instance Connect could not verify the signer trust chain. No keys have been trusted.
C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
error 89 at 4 depth lookup: Basic Constraints of CA cert not marked critical
C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
error 92 at 4 depth lookup: CA cert does not include key usage extension
error /dev/shm/eic-l8wtfU3e/cert.pem: verification failed
```

It seem like the issue is either with `mssh` generating invalid keys or with `ec2-instance-connect` performing a trust verification that is too strict.

Contributor guide

Open the contributing guide

Research direction

Start with /usr/share/ec2-instance-connect/eic_parse_authorized_keys and the eic_run_authorized_keys invocation shown in the instance log. Reproduce the failure on Ubuntu 22.04 with mssh --debug, then inspect the OpenSSL trust-chain verification output; done means the signer chain is handled correctly and the connection no longer ends with Permission denied (publickey).

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, shell
Domain
authentication, operating-systems, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.