aws / aws/amazon-ssm-agent

ssm agent failing to start after reboot

Open
#320 21 comments 6 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.2k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

We are applying patches to our Windows instances using the patch manager function in AWS Systems Manager. We have a patch baseline that is executed against a set of windows instances (each of which are part of a patch group) by executing a maintenance window which in turn executes a run command(AWS-RunPatchBaseline) against each of the instances. However we are finding the following:

The instances in question seem to get patches installed correctly. Executing `wmic qfe list` shows that the patches have been installed on the target machines
The target instances are then rebooted after patches are installed
The run command remains in progress indefinitely
From more investigation we found that the amazon-ssh-agent failed to start when the instances are rebooted. Looking at event logs shows a timeout occured:

`Get-WinEvent -ProviderName 'Service Control Manager'`

Output:
```
09/11/2020 14:25:56 7000 Error The AmazonSSMAgent service failed to start due to the following error: …
09/11/2020 14:25:56 7009 Error A timeout was reached (30000 milliseconds) while waiting for the AmazonSSMAgent service to connect.
```

Once we manually restarted the amazon-ssh-agent again the run command completed successfully. This issue is we do not want to have to manually start the amazon-ssh-agent on each instance especially as we have a lot of instances.
This suggests that it is not an issue with Persistent Routes either and I have just double checked:

`Instance IP 10.1.3.217`

Persistent Routes:
```
Network Address Netmask Gateway Address Metric
169.254.169.254 255.255.255.255 10.1.3.1 15
169.254.169.250 255.255.255.255 10.1.3.1 15
169.254.169.251 255.255.255.255 10.1.3.1 15
169.254.169.249 255.255.255.255 10.1.3.1 15
169.254.169.123 255.255.255.255 10.1.3.1 15
169.254.169.253 255.255.255.255 10.1.3.1 15
```

Any ideas on what is causing this, i.e. why is the amazon-ssh-agent not starting up successfully after automatic reboot?

Contributor guide

Open the contributing guide

Research direction

Start with the Windows Service Control Manager events 7000 and 7009 and the AWS-RunPatchBaseline reboot flow described in the issue. Reproduce the automatic reboot on a patched Windows instance and compare it with a manual AmazonSSMAgent restart; done means the agent starts after reboot and the run command completes without manual intervention.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, devops, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.