hashicorp / hashicorp/go-plugin

Logging and SIGPIPE when host process dies

Open
#114 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
6.1k
Forks
511
Avg merge
22h 47m
Merged PRs (30d)
4

Description

Nomad uses go-plugin to spin up various plugins and auxiliary processes, and saw surprising (to us) behavior when host process dies in https://github.com/hashicorp/nomad/pull/5598 .

Nomad uses go-plugin to spin up long-running plugins with lifecycle independent from host, to ease in-place upgrades and reconfiguration, and use the reattachment patterns `ReattachConfig` supported by this.

However, we observe the following problems after host process is restarted:
1. The plugin gets a SIGPIPE signal upon the next log/Stdout/Stderr write operation. When the host (e.g. go-plugin client) process dies, Stdout/Stderr pipe closes and any write from plugin fails with `io.ErrClosedPipe` error, and the plugin receive SIGPIPE, typically killing it.
* note that if plugin explicitly ignores `SIGPIPE`, hclog may panics on log write failure in https://github.com/hashicorp/go-hclog/blob/6907afbebd2eef854f0be9194eb79b0ba75d7b29/intlogger.go#L370-L373

2. On successful reattachment by a restarted host process, stdout/stderr syncing is lost, and any plugin log lines to Stdout/Stderr are lost.

Nomad works around this by having a dedicated log file for the plugin and not writing to the plugin Stderr in https://github.com/hashicorp/nomad/pull/5598 .

Ideally, `go-plugin` can makes handling host process restarting and re-attaching better. One possibility might be using fifo files such that plugin can always write to it with some buffer, but this may require clever use of non-blocking flags (to ensure plugin can proceed when fifo buffer is full).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.