aws / aws/aws-cli

ec2 instance connect fails sporadically on WebSocket (NO_PROXY=1 does not help)

Open
#8,880 5 comments 1 reaction 0 assignees View on GitHub
bug ec2-instance-connect p2
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the bug

aws ec2-instance-connect open-tunnel

Fails with
Exception in WebSocket on_connection_setup callback
> Traceback (most recent call last):
> File "awscrt\websocket.py", line 459, in _on_connection_setup
> File "awscli\customizations\ec2instanceconnect\websocket.py", line 239, in _on_connection
> TypeError: 'NoneType' object is not iterable

and is stuck.

I was unable to detect any recurring pattern. It could happen on the first connection or reconnection attempt in case of a temporal network disconnect.

### Expected Behavior

Either do not fail at all, or fail quickly allowing an automatic recover.

### Current Behavior

It is stuck and I need to restart VSCode to get rid of it.

### Reproduction Steps

It happens within my custom PowrShell script presented below:
# Initiate an SSH session over AWS EC2 Instance Connect Endpoint using a temporary Public Key.

param (
[string]$HostName, # Assume $input is the input string "/"
[int]$PortNumber
)

# Global configuration.
$PROFILE_NAME = $HostName.Split('/')[0]
$INSTANCE_NAME_TAG = $HostName.Split('/')[1]
$SSH_DIR = "$env:USERPROFILE\.ssh"
$SSH_TMP_KEY = "$SSH_DIR\ssh-tmp"
$SSH_USER = "ec2-user"
$env:NO_PROXY="1"

# Remove expired ssh private and public keys
function Clear-SSHKeyCache {
Write-Host "Cleaning up the SSH Keys Cache"
Remove-Item "$SSH_TMP_KEY*" -Force
& ssh-add -D
}

# Log in to AWS SSO, if the session expired
function Connect-AWSSSOLogin {
# Attempt to get the AWS caller identity to check if the SSO session is valid.
aws sts get-caller-identity --profile $PROFILE_NAME *>&1
if ($LASTEXITCODE -ne 0) {
Write-Host "SSO session expired or not valid"
Clear-SSHKeyCache
Write-Host "Logging in ..."
if (aws sso login --profile $PROFILE_NAME) {
Write-Host "SSO login successful."
} else {
Write-Host "SSO login failed"
exit 1
}
} else {
Write-Host "SSO session is still valid, using existing credentials."
}
}

# Find EC2 Instance ID by name
function Find-InstanceID {
$INSTANCE_ID = aws ec2 describe-instances --profile $PROFILE_NAME `
--filters "Name=tag:Name,Values=$INSTANCE_NAME_TAG" `
"Name=instance-state-name,Values=pending,running,stopping,stopped" `
--query "Reservations[*].Instances[*].InstanceId" --output text

if (-not $INSTANCE_ID) {
Write-Host "No non-terminated instances found with the Name tag: $INSTANCE_NAME_TAG"
exit 1
}

Write-Host "Instance ID: $INSTANCE_ID"
return $INSTANCE_ID
}

# Wait for a particular instance state
function Wait-ForInstanceState($INSTANCE_ID, $STATE) {
$state = "instance-$STATE"
Write-Host "Waiting for instance to enter $state state..."
aws ec2 wait $state --profile $PROFILE_NAME --instance-ids $INSTANCE_ID
}

# Start the instance and wait for it to become running
function Start-AndWaitInstance($INSTANCE_ID) {
Write-Host "Starting instance $INSTANCE_ID ..."
aws ec2 start-instances --profile $PROFILE_NAME --instance-ids $INSTANCE_ID
Wait-ForInstanceState $INSTANCE_ID "running"
}

# Checks and manages instance state.
function Test-AndManageInstanceState($INSTANCE_ID) {
$INSTANCE_STATE = aws ec2 describe-instance-status --profile $PROFILE_NAME `
--instance-id $INSTANCE_ID --include-all-instances `
--query InstanceStatuses[0].InstanceState.Name --output text

switch ($INSTANCE_STATE) {
"stopped" {
Write-Host "Instance is stopped."
Start-AndWaitInstance $INSTANCE_ID
}
"pending" {
Write-Host "Instance is pending"
Wait-ForInstanceState $INSTANCE_ID "running"
}
"stopping" {
Write-Host "Instance is stopping..."
Wait-ForInstanceState $INSTANCE_ID "stopped"
Start-AndWaitInstance $INSTANCE_ID
}
"running" {
Write-Host "Instance $INSTANCE_ID is already running."
}
default {
Write-Host "Unexpected instance state: $INSTANCE_STATE"
exit 1
}
}
}

# Generates a temporary SSH key.
function New-SSHKey {
if (-not (Test-Path $SSH_TMP_KEY)) {
Write-Host "Generating a temporary SSH key..."
ssh-keygen -t rsa -b 2048 -f $SSH_TMP_KEY -N '""' *> $null 2>&1
Set-ItemProperty -Path $SSH_TMP_KEY -Name IsReadOnly -Value $true
}
$SSH_PUB_KEY = Get-Content "$SSH_TMP_KEY.pub" -Raw
return $SSH_PUB_KEY
}

# Sends the public key to the instance via EIC.
function Send-PublicKey($INSTANCE_ID, $SSH_PUB_KEY) {
Write-Host "Sending public key to the instance $INSTANCE_ID ..."
$response = aws ec2-instance-connect send-ssh-public-key `
--instance-id $INSTANCE_ID `
--profile $PROFILE_NAME `
--instance-os-user $SSH_USER `
--ssh-public-key $SSH_PUB_KEY
Write-Host "send-ssh-public-key response $response"
}

# Starts SSH session.
function Open-Tunnel($INSTANCE_ID) {
Write-Host "Opening tunnel over EIC..."
aws ec2-instance-connect open-tunnel `
--instance-id $INSTANCE_ID `
--profile $PROFILE_NAME
}

# Main script execution.
Connect-AWSSSOLogin
$INSTANCE_ID = Find-InstanceID
Test-AndManageInstanceState $INSTANCE_ID
$SSH_PUB_KEY = New-SSHKey
Send-PublicKey $INSTANCE_ID $SSH_PUB_KEY
Open-Tunnel $INSTANCE_ID

I normally use it from VSCode, but the same problem happens with ssh command. I never experienced such a problem on my Linux client desktop, only Windows.

### Possible Solution

One possible explanation is a time delay after an automatic instance restart. If I had a way to whether EIC is available and to wait until it is, that would solve the problem. Alternatively, fail fast in order to retry.

### Additional Information/Context

_No response_

### CLI version used

aws-cli/2.17.27 Python/3.11.9 Windows/10 exe/AMD64

### Environment details (OS name and version, etc.)

Windows 10, PowerShell 7.4.4

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.