hashicorp / hashicorp/go-plugin
Logging and SIGPIPE when host process dies
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 511
- Avg merge
- 22h 47m
- Merged PRs (30d)
- 4
Description
Nomad uses go-plugin to spin up various plugins and auxiliary processes, and saw surprising (to us) behavior when host process dies in https://github.com/hashicorp/nomad/pull/5598 .
Nomad uses go-plugin to spin up long-running plugins with lifecycle independent from host, to ease in-place upgrades and reconfiguration, and use the reattachment patterns `ReattachConfig` supported by this.
However, we observe the following problems after host process is restarted:
1. The plugin gets a SIGPIPE signal upon the next log/Stdout/Stderr write operation. When the host (e.g. go-plugin client) process dies, Stdout/Stderr pipe closes and any write from plugin fails with `io.ErrClosedPipe` error, and the plugin receive SIGPIPE, typically killing it.
* note that if plugin explicitly ignores `SIGPIPE`, hclog may panics on log write failure in https://github.com/hashicorp/go-hclog/blob/6907afbebd2eef854f0be9194eb79b0ba75d7b29/intlogger.go#L370-L373
2. On successful reattachment by a restarted host process, stdout/stderr syncing is lost, and any plugin log lines to Stdout/Stderr are lost.
Nomad works around this by having a dedicated log file for the plugin and not writing to the plugin Stderr in https://github.com/hashicorp/nomad/pull/5598 .
Ideally, `go-plugin` can makes handling host process restarting and re-attaching better. One possibility might be using fifo files such that plugin can always write to it with some buffer, but this may require clever use of non-blocking flags (to ensure plugin can proceed when fifo buffer is full).
Contributor guide
Assessment
This issue has not been assessed yet.