aws / aws/aws-ec2-instance-connect-config

ec2-instance-connect v1.1.12 broke our existing SSH Config

Open
#19 8 comments 11 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
88
Forks
39
PR merge metrics
No merged PRs in 30d

Description

# Issue Summary
When we upgraded the `ec2-instance-connect` package from 1.1.9 to 1.1.12 on Ubuntu 16.04 and 18.04, ec2-instance-connect overrode our `AuthorizedKeysCommand` and `AuthorizedKeysCommandUser`.

For our use case we want to support two sets of `AuthorizedKeysCommand`

1. one for a directory backed system and
2. one for EC2 instance connect when certain users are logging in

I believe it was [this commit](https://github.com/aws/aws-ec2-instance-connect-config/commit/99b94d6596ae93ca416b4a9f96a30dac9c7cc0e4) which corrected the systemd service file that introduced this issue for us.

## Our config
Here is what our `sshd_config` looked like (some values replaced).

```
AuthorizedKeysCommand /usr/bin/
AuthorizedKeysCommandUser nobody
AllowGroups
AuthenticationMethods publickey

Match Group ec2i
AllowGroups
AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys %u %f
AuthorizedKeysCommandUser ec2-instance-connect
```
## `auth.log`

When attempting our default login path (non-ec2i) we could see that sshd was attempting to use the ec2 instance connect commands to look up authorized keys.

```
2020-03-02T16:34:40.896366+00:00 ip-192-168-1-1 sshd[1234]: Connection closed by authenticating user my_user 1.1.1.1 port 63090 [preauth]
2020-03-02T16:35:37.982098+00:00 ip-192-168-1-1 sshd[4567]: AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys my_user SHA256:Ahje+...
```

## Service details
```
$ sudo service sshd status
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/ssh.service.d
└─ec2-instance-connect.conf
Active: active (running) since Mon 2020-03-02 20:51:48 UTC; 1min 13s ago
Main PID: 1744 (sshd)
Tasks: 9
Memory: 141.8M
CPU: 2.828s
CGroup: /system.slice/ssh.service
├─1744 /usr/sbin/sshd -D -o AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys %u %f -o AuthorizedKeysCommandUser ec2-instance-con
....
```

# Solution

We have _downgraded and pinned_ version 1.1.9. This restored our ability to SSH via our directory backed `AuthorizedKeysCommand`.

## Downgrade service details
```
$ sudo service sshd status
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-03-02 17:43:31 UTC; 3s ago
Process: 8539 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 8540 (sshd)
Tasks: 8 (limit: 4915)
CGroup: /system.slice/ssh.service
├─8540 /usr/sbin/sshd -D
....
```

# Suggestions
1. Do not pass in explicit config settings to `sshd` through [the service config](https://github.com/aws/aws-ec2-instance-connect-config/blob/master/src/deb_systemd/ssh.service.d/ec2-instance-connect.conf)
2. Write a test to ensure that any existing `AuthorizedKeysCommand` is respected.
3. Don't attempt to make any changes to the `AuthorizedKeysCommand` on install. Instead print a message that tells users how to configure `sshd_config` for themselves.

Contributor guide

Open the contributing guide

Research direction

Inspect src/deb_systemd/ssh.service.d/ec2-instance-connect.conf and compare the service behavior between versions 1.1.9 and 1.1.12. Verify how an existing AuthorizedKeysCommand is handled and add coverage for that configuration; done means the EC2 Instance Connect service no longer overrides the existing command for non-EC2-instance-connect users.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, shell, ubuntu
Domain
infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.