hashicorp / hashicorp/packer-plugin-amazon

Use SSM ResumeSession API to support EC2 instance reboot via SSM session-manager-plugin

Open
#699 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
91
Forks
141
Avg merge
2d 18h
Merged PRs (30d)
3

Description

#### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

Please search the existing issues for relevant feature requests, and use the
reaction feature
(https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/)
to add upvotes to pre-existing requests.

#### Description

Currently, Packer building via "session-manager-plugin" (AWS SSM Session Manager) will report several similar, related error message(s). They are **cosmetic** and do not affect the resulting AMI, but it generates wastage in terms of the SSM session having to be re-created (potentially multiple times in my case) and the eventual session will always fail to be terminated as a result.

Some example error messages:

```bash
==> amazon-ebs.: Provisioning with shell script: /tmp/packer-shell896848410
==> amazon-ebs.: SessionId: : document process failed unexpectedly: ipc messaging received timeout signal , check [ssm-document-worker]/[ssm-session-worker] log for crash reason
==> amazon-ebs.: Error terminating SSM Session "", this does not affect the built AMI. Please terminate the session manually: operation error SSM: TerminateSession, https response error StatusCode: 400, RequestID: ad72a001-b511-4bbb-bc88-f899966fe7a2, api error ValidationException: Session is not in a valid state

==> amazon-ebs.: Waiting for AMI to become ready...
==> amazon-ebs.: SessionId: : document process failed unexpectedly: ipc messaging received timeout signal , check [ssm-document-worker]/[ssm-session-worker] log for crash reason
==> amazon-ebs.: Error terminating SSM Session "", this does not affect the built AMI. Please terminate the session manually: operation error SSM: TerminateSession, https response error StatusCode: 400, RequestID: 0b8c0f68-e2b3-4ccb-a540-1518e5ff39de, api error ValidationException: Session is not in a valid state
```

These messages are caused by the build instance restarting. The AWS SSM agent currently does not survive a restart. It tears down the port forwarding session, the `session-manager-plugin` reports the loss, and Packer's follow-up `ssm:TerminateSession` call is rejected because the session is already terminating. Packer then opens a replacement session and continues.

To help a reader understand the order of events and why Packer should be enhanced to support the build instance being 'rebooted' can be seen in this image:

Image

^In the above image, you can see that I need to compile/install a custom kernel, requiring a machine reboot before I proceed with another Packer provisioner with downstream further config/installation that I must carry out as part of the Packer build process. I'm also using EC2 Spot to save cost and this needs a reboot as well during the AMI creation process. So I get a 2nd session failure as a result of the 2nd reboot and the new tunnel leaks, so the terminate session fails. I currently have to run a post-processor with "shell-local" to execute a cleanup of this SSM session.

All of this could be fixed in Packer... :)

#### Use Case(s)

The Packer SSM communicator should handle reboots gracefully by creating a new session after the instance comes back up (rather than expecting the existing session to survive, which is not supported now).

Packer possibly needs to:
   1. Detect the session died due to reboot (vs. a real failure)
   2. Wait for the instance to pass SSM connectivity checks again
   3. Try to start a new session

#### Potential Solution

So, this is the important bit....

**session-manager-plugin** v1.2.835.0 added this feature for client-side automatic reconnection: https://github.com/aws/session-manager-plugin/releases/tag/1.2.835.0

> Fix port forwarding sessions silently dying after WebSocket reconnection

and if you read the commit notes here: https://github.com/aws/session-manager-plugin/commit/29db6846b35f50feb5713199a1118ad2ac756df7

> Fix data channel reconnection: use StreamUrl and refresh credentials
> Fix long-lived port forwarding sessions silently dying after WebSocket
reconnection. GetResumeSessionParams() now returns and propagates the
fresh StreamUrl, and ResumeSessionHandler() updates credentials on the
WebSocket channel before reconnecting.

What we are missing is integration in Packer to:

* Store url/token on successful session connection
* Detect due to reboot. Do we even need to do this, as standard SSH auto-recovers nicely in Packer already.
* Resume Session API
* Build websocket with returned url and token (from failed session)
* Once data channel is reconnected, the session can come back
* add configurable timeout only applicable when using SSM/session-manager-plugin

and thus, we no longer see all these horrible **ERROR** messages when the build instance is rebooted and/or restarted for AMI creation (when EC2 Spot is used and Packer does not pass `NoReboot` for consistent snapshot).

This would then bring the customer experience inline with standard SSH communicator during an Ec2 instance reboot and/or `NoReboot` of `ec2:CreateImage` AMI creation (where the Ec2 Spot instance is rebooted).

#### Potential References

Advice from AWS SDE expert on SSM Session Manager on how to achieve this:
https://github.com/aws/session-manager-plugin/issues/55#issuecomment-1412644170

Contributor guide

Open the contributing guide

Research direction

The issue does not name implementation files, tests, or entry points. Start by locating the Amazon SSM communicator and its session-manager-plugin connection lifecycle, then determine how ResumeSession, reconnection detection, and timeout configuration fit together; done means reboots reconnect cleanly without leaked sessions or termination errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.