ec2 instance connect fails sporadically on WebSocket (NO_PROXY=1 does not help)
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
### Describe the bug
aws ec2-instance-connect open-tunnel
Fails with
Exception in WebSocket on_connection_setup callback
> Traceback (most recent call last):
> File "awscrt\websocket.py", line 459, in _on_connection_setup
> File "awscli\customizations\ec2instanceconnect\websocket.py", line 239, in _on_connection
> TypeError: 'NoneType' object is not iterable
and is stuck.
I was unable to detect any recurring pattern. It could happen on the first connection or reconnection attempt in case of a temporal network disconnect.
### Expected Behavior
Either do not fail at all, or fail quickly allowing an automatic recover.
### Current Behavior
It is stuck and I need to restart VSCode to get rid of it.
### Reproduction Steps
It happens within my custom PowrShell script presented below:
# Initiate an SSH session over AWS EC2 Instance Connect Endpoint using a temporary Public Key.
param (
[string]$HostName, # Assume $input is the input string "/"
[int]$PortNumber
)
# Global configuration.
$PROFILE_NAME = $HostName.Split('/')[0]
$INSTANCE_NAME_TAG = $HostName.Split('/')[1]
$SSH_DIR = "$env:USERPROFILE\.ssh"
$SSH_TMP_KEY = "$SSH_DIR\ssh-tmp"
$SSH_USER = "ec2-user"
$env:NO_PROXY="1"
# Remove expired ssh private and public keys
function Clear-SSHKeyCache {
Write-Host "Cleaning up the SSH Keys Cache"
Remove-Item "$SSH_TMP_KEY*" -Force
& ssh-add -D
}
# Log in to AWS SSO, if the session expired
function Connect-AWSSSOLogin {
# Attempt to get the AWS caller identity to check if the SSO session is valid.
aws sts get-caller-identity --profile $PROFILE_NAME *>&1
if ($LASTEXITCODE -ne 0) {
Write-Host "SSO session expired or not valid"
Clear-SSHKeyCache
Write-Host "Logging in ..."
if (aws sso login --profile $PROFILE_NAME) {
Write-Host "SSO login successful."
} else {
Write-Host "SSO login failed"
exit 1
}
} else {
Write-Host "SSO session is still valid, using existing credentials."
}
}
# Find EC2 Instance ID by name
function Find-InstanceID {
$INSTANCE_ID = aws ec2 describe-instances --profile $PROFILE_NAME `
--filters "Name=tag:Name,Values=$INSTANCE_NAME_TAG" `
"Name=instance-state-name,Values=pending,running,stopping,stopped" `
--query "Reservations[*].Instances[*].InstanceId" --output text
if (-not $INSTANCE_ID) {
Write-Host "No non-terminated instances found with the Name tag: $INSTANCE_NAME_TAG"
exit 1
}
Write-Host "Instance ID: $INSTANCE_ID"
return $INSTANCE_ID
}
# Wait for a particular instance state
function Wait-ForInstanceState($INSTANCE_ID, $STATE) {
$state = "instance-$STATE"
Write-Host "Waiting for instance to enter $state state..."
aws ec2 wait $state --profile $PROFILE_NAME --instance-ids $INSTANCE_ID
}
# Start the instance and wait for it to become running
function Start-AndWaitInstance($INSTANCE_ID) {
Write-Host "Starting instance $INSTANCE_ID ..."
aws ec2 start-instances --profile $PROFILE_NAME --instance-ids $INSTANCE_ID
Wait-ForInstanceState $INSTANCE_ID "running"
}
# Checks and manages instance state.
function Test-AndManageInstanceState($INSTANCE_ID) {
$INSTANCE_STATE = aws ec2 describe-instance-status --profile $PROFILE_NAME `
--instance-id $INSTANCE_ID --include-all-instances `
--query InstanceStatuses[0].InstanceState.Name --output text
switch ($INSTANCE_STATE) {
"stopped" {
Write-Host "Instance is stopped."
Start-AndWaitInstance $INSTANCE_ID
}
"pending" {
Write-Host "Instance is pending"
Wait-ForInstanceState $INSTANCE_ID "running"
}
"stopping" {
Write-Host "Instance is stopping..."
Wait-ForInstanceState $INSTANCE_ID "stopped"
Start-AndWaitInstance $INSTANCE_ID
}
"running" {
Write-Host "Instance $INSTANCE_ID is already running."
}
default {
Write-Host "Unexpected instance state: $INSTANCE_STATE"
exit 1
}
}
}
# Generates a temporary SSH key.
function New-SSHKey {
if (-not (Test-Path $SSH_TMP_KEY)) {
Write-Host "Generating a temporary SSH key..."
ssh-keygen -t rsa -b 2048 -f $SSH_TMP_KEY -N '""' *> $null 2>&1
Set-ItemProperty -Path $SSH_TMP_KEY -Name IsReadOnly -Value $true
}
$SSH_PUB_KEY = Get-Content "$SSH_TMP_KEY.pub" -Raw
return $SSH_PUB_KEY
}
# Sends the public key to the instance via EIC.
function Send-PublicKey($INSTANCE_ID, $SSH_PUB_KEY) {
Write-Host "Sending public key to the instance $INSTANCE_ID ..."
$response = aws ec2-instance-connect send-ssh-public-key `
--instance-id $INSTANCE_ID `
--profile $PROFILE_NAME `
--instance-os-user $SSH_USER `
--ssh-public-key $SSH_PUB_KEY
Write-Host "send-ssh-public-key response $response"
}
# Starts SSH session.
function Open-Tunnel($INSTANCE_ID) {
Write-Host "Opening tunnel over EIC..."
aws ec2-instance-connect open-tunnel `
--instance-id $INSTANCE_ID `
--profile $PROFILE_NAME
}
# Main script execution.
Connect-AWSSSOLogin
$INSTANCE_ID = Find-InstanceID
Test-AndManageInstanceState $INSTANCE_ID
$SSH_PUB_KEY = New-SSHKey
Send-PublicKey $INSTANCE_ID $SSH_PUB_KEY
Open-Tunnel $INSTANCE_ID
I normally use it from VSCode, but the same problem happens with ssh command. I never experienced such a problem on my Linux client desktop, only Windows.
### Possible Solution
One possible explanation is a time delay after an automatic instance restart. If I had a way to whether EIC is available and to wait until it is, that would solve the problem. Alternatively, fail fast in order to retry.
### Additional Information/Context
_No response_
### CLI version used
aws-cli/2.17.27 Python/3.11.9 Windows/10 exe/AMD64
### Environment details (OS name and version, etc.)
Windows 10, PowerShell 7.4.4
Contributor guide
Assessment
This issue has not been assessed yet.