aws / aws/aws-cli

ec2-instance-connect open-tunnel doesn't exit after pipe is closed

Open
#9,344 1 comment 2 reactions 1 assignee Claimed by @RyanFitzSimmonsAK View on GitHub
bug customization ec2-instance-connect p3 service-api
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the bug

When using `aws ec2-instance-connect open-tunnel` as a pipe, after a successful connection happens and parent process closes the pipe, the `aws` process continues to linger and uses a surprising amount of CPU. When attaching a debugger to the process, it appears stuck on a `futex` syscall.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

The `aws` process should terminate gracefully once the parent connection closes the pipe and exits.

### Current Behavior

The process lingers after the parent process exits. (Note high CPU usage)
```
matt 93039 84.1 0.2 631504 87456 pts/6 SNl 13:10 0:03 aws --debug --region us-east-1 ec2-instance-connect open-tunnel --instance-id i-xxxx --private-ip-address 172.x.x.x --instance-connect-endpoint-id eice-xxxx --max-tunnel-duration 3600
```

In debug mode, it logs a lot of this to stderr in this state.
```
[DEBUG] [2025-03-05T13:11:22Z] [00007e1d716006c0] [websocket] - id=0x7e1d64017240: Enqueuing outgoing frame with opcode=2(binary) length=0 fin=T
[DEBUG] [2025-03-05T13:11:22Z] [00007e1d716006c0] [websocket] - id=0x7e1d64017240: Enqueuing outgoing frame with opcode=2(binary) length=0 fin=T
[DEBUG] [2025-03-05T13:11:22Z] [00007e1d716006c0] [websocket] - id=0x7e1d64017240: Enqueuing outgoing frame with opcode=2(binary) length=0 fin=T
[DEBUG] [2025-03-05T13:11:22Z] [00007e1d716006c0] [websocket] - id=0x7e1d64017240: Enqueuing outgoing frame with opcode=2(binary) length=0 fin=T
[DEBUG] [2025-03-05T13:11:22Z] [00007e1d716006c0] [websocket] - id=0x7e1d64017240: Enqueuing outgoing frame with opcode=2(binary) length=0 fin=T
[DEBUG] [2025-03-05T13:11:22Z] [00007e1d716006c0] [websocket] - id=0x7e1d64017240: Enqueuing outgoing frame with opcode=2(binary) length=0 fin=T
[DEBUG] [2025-03-05T13:11:22Z] [00007e1d716006c0] [websocket] - id=0x7e1d64017240: Enqueuing outgoing frame with opcode=2(binary) length=0 fin=T
```

Attaching to the process, we see:
```
strace: Process 93039 attached
futex(0x7ad7bf0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY
```

Also looking at `ss -np` output, we can see AWS is still holding open established TCP connections for the websockets.

### Reproduction Steps

I haven't found a general reproduction case outside of what I was attempting to use this command for. I was adding proxy_command support to Hashicorp's Terraform communicator (https://github.com/hashicorp/terraform/pull/36643). Basically, if the aws process doesn't get a signal to cleanup, it will still continue to run with the websocket active. I got the desired behavior by ensuring the aws process receives a SIGHUP when cleaning up the connection but I also got the desired behavior by adding more cleanup to the websocket code in aws. PR forthcoming.

### Possible Solution

Implement more cleanup to the websocket code.

### Additional Information/Context

_No response_

### CLI version used

2.24.17

### Environment details (OS name and version, etc.)

Ubuntu 22.04.2

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.