hashicorp / hashicorp/packer-plugin-amazon
amazon-ebs v1.8.1: SSH stdout relay ~50-70x slower than v1.8.0; verbose provisioners time out with "exit status: 123"
- Dominant language
- Go
- Stars
- 91
- Forks
- 141
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 3
Description
#### Overview of the Issue
After the amazon plugin auto-upgraded from **v1.8.0 → v1.8.1**, our `amazon-ebs` builds began failing with:
```
Build 'amazon-ebs.xxx' errored after N minutes: Script exited with non-zero exit status: 123. Allowed exit codes are: [0]
```
The remote provisioner scripts actually **complete successfully** — the `123` is the SDK's lost-channel sentinel (`packer-plugin-sdk/rpc/communicator.go` `Start()` calls `cmd.SetExited(123)` when it can't receive the command's exit status over the response stream).
Root of it: **v1.8.1 relays remote stdout over the SSH communicator far slower than v1.8.0.** We measured the per-line emission delay on identical provisioner output:
| plugin | median delay per output line | a ~1050-line provisioner step |
|---|---|---|
| v1.8.0 | ~0.007 s | **~20 s** |
| v1.8.1 | ~0.48–0.62 s | **~535 s (≈9 min)** |
Same Packer core, same runner, same OpenSSH client, same guest AMIs — only the plugin version differs. Because the slowdown is per-line, any provisioner that emits a lot of output (ours print ~1000–1100 lines) drags from seconds to many minutes, and the SSH/RPC channel eventually drops mid-stream → `exit status: 123`. **Pinning back to v1.8.0 fully restores normal speed and green builds.**
We suspect v1.8.1's dependency bumps in the SSH/communicator path — `golang.org/x/crypto` (v0.43 → v0.52) and `packer-plugin-sdk` — but we have **not** bisected to a specific commit.
#### Reproduction Steps
1. An `amazon-ebs` build whose shell provisioner prints ~1000+ stdout lines (e.g. a loop that echoes per-item progress).
2. Build with plugin **v1.8.0** → output streams instantly, build succeeds.
3. Build with plugin **v1.8.1** (no other change) → each output line surfaces ~0.5 s apart; the verbose step takes minutes; the build ends with `Script exited with non-zero exit status: 123` even though the script finished.
### Plugin and Packer version
- Packer **v1.15.3** (unchanged across both runs)
- `github.com/hashicorp/amazon` **v1.8.0** (works) vs **v1.8.1** (fails)
### Simplified Packer Buildfile
`source "amazon-ebs"` with the default SSH communicator (no special `ssh_*` options). The trigger is simply a provisioner that produces a large amount of stdout; no unusual builder config is required.
### Operating system and Environment details
- Builder: GitHub-hosted `ubuntu-24.04` runner (x86_64), OpenSSH client `1:9.6p1-3ubuntu13.16`.
- Build instances: EC2 `m6a.large` in `us-west-2`, public IP via default VPC, cross-region/cross-network SSH from the runner.
- Reproduced identically on two different guest OSes (Ubuntu 22.04 and Amazon Linux 2023), so it is not guest-specific.
### Log Fragments and crash.log files
The visible signature in the build log is the provisioner's progress lines surfacing ~0.5 s apart on v1.8.1 (vs near-instant on v1.8.0), followed by `Script exited with non-zero exit status: 123` shortly after the script's final line prints. Happy to attach `PACKER_LOG=1` logs or test a candidate fix.
Contributor guide
Assessment
This issue has not been assessed yet.