hashicorp / hashicorp/packer-plugin-amazon

amazon-ebs v1.8.1: SSH stdout relay ~50-70x slower than v1.8.0; verbose provisioners time out with "exit status: 123"

Open
#678 12 comments 30 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
91
Forks
141
Avg merge
2d 18h
Merged PRs (30d)
3

Description

#### Overview of the Issue

After the amazon plugin auto-upgraded from **v1.8.0 → v1.8.1**, our `amazon-ebs` builds began failing with:

```
Build 'amazon-ebs.xxx' errored after N minutes: Script exited with non-zero exit status: 123. Allowed exit codes are: [0]
```

The remote provisioner scripts actually **complete successfully** — the `123` is the SDK's lost-channel sentinel (`packer-plugin-sdk/rpc/communicator.go` `Start()` calls `cmd.SetExited(123)` when it can't receive the command's exit status over the response stream).

Root of it: **v1.8.1 relays remote stdout over the SSH communicator far slower than v1.8.0.** We measured the per-line emission delay on identical provisioner output:

| plugin | median delay per output line | a ~1050-line provisioner step |
|---|---|---|
| v1.8.0 | ~0.007 s | **~20 s** |
| v1.8.1 | ~0.48–0.62 s | **~535 s (≈9 min)** |

Same Packer core, same runner, same OpenSSH client, same guest AMIs — only the plugin version differs. Because the slowdown is per-line, any provisioner that emits a lot of output (ours print ~1000–1100 lines) drags from seconds to many minutes, and the SSH/RPC channel eventually drops mid-stream → `exit status: 123`. **Pinning back to v1.8.0 fully restores normal speed and green builds.**

We suspect v1.8.1's dependency bumps in the SSH/communicator path — `golang.org/x/crypto` (v0.43 → v0.52) and `packer-plugin-sdk` — but we have **not** bisected to a specific commit.

#### Reproduction Steps

1. An `amazon-ebs` build whose shell provisioner prints ~1000+ stdout lines (e.g. a loop that echoes per-item progress).
2. Build with plugin **v1.8.0** → output streams instantly, build succeeds.
3. Build with plugin **v1.8.1** (no other change) → each output line surfaces ~0.5 s apart; the verbose step takes minutes; the build ends with `Script exited with non-zero exit status: 123` even though the script finished.

### Plugin and Packer version

- Packer **v1.15.3** (unchanged across both runs)
- `github.com/hashicorp/amazon` **v1.8.0** (works) vs **v1.8.1** (fails)

### Simplified Packer Buildfile

`source "amazon-ebs"` with the default SSH communicator (no special `ssh_*` options). The trigger is simply a provisioner that produces a large amount of stdout; no unusual builder config is required.

### Operating system and Environment details

- Builder: GitHub-hosted `ubuntu-24.04` runner (x86_64), OpenSSH client `1:9.6p1-3ubuntu13.16`.
- Build instances: EC2 `m6a.large` in `us-west-2`, public IP via default VPC, cross-region/cross-network SSH from the runner.
- Reproduced identically on two different guest OSes (Ubuntu 22.04 and Amazon Linux 2023), so it is not guest-specific.

### Log Fragments and crash.log files

The visible signature in the build log is the provisioner's progress lines surfacing ~0.5 s apart on v1.8.1 (vs near-instant on v1.8.0), followed by `Script exited with non-zero exit status: 123` shortly after the script's final line prints. Happy to attach `PACKER_LOG=1` logs or test a candidate fix.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.