Starting in agent v3.29.0, bootstrap-script on Windows must be an absolute path
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 378
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 74
Description
CC @wesleyw72 @pzeballos
We have discovered that due to [a change](https://github.com/buildkite/agent/pull/1402) included in agent v3.29.0, Windows agents can no longer specify `bootstrap-script` with an executable on the `PATH`, and must now specify an absolute path.
In our case, this is a problem because we use `bash` to wrap our `bootstrap-script`, and we specify the value of this configuration setting like (for example): `bootstrap-script="bash c:/users/buildkite-agent/AppData/Local/buildkite-agent/hooks/bootstrap"`. This previously worked, but when we try this exact same configuration on v3.29.0 or newer, this fails with the error message `exec: "c:\\b\\bash.exe": file does not exist`.
We bisected the agent changes and narrowed it down to the aforementioned PR - if we build a copy of v3.29.0 without that single PR, then this bug does not exist.
We have further traced this behavior down to a strange behavior inside golang's `os.exec` package - possibly a golang bug? - where on Windows, if you have an `os.exec.Cmd` struct where `Dir` is set, Windows PATH resolution is disabled, and the value supplied as `PATH` must be an absolute path to the executable.
See [the code here](https://golang.org/src/os/exec/exec.go) in `lookExtensions()` for the exact logic used - note where if `Dir` is not set, the `PATH` is searched for `path`, whereas if `Dir` is set, the code attempts to do a `PATH` search for `dir+path` instead, which can only ever match either a fully-qualified absolute path or an executable in `CWD`.
We have a workaround in that we can change our agent configuration to use the absolute path for `bash` in `bootstrap-script`, but this is a highly unexpected behavior, especially since it appeared out of nowhere, was highly non-obvious behavior, the error messages returned are not particularly helpful in figuring this out, and this behavior does not exist on any other platform.
Contributor guide
Research direction
Start with the bootstrap-script execution path affected by PR 1402 and compare agent v3.29.0 with the prior behavior on Windows. Review Go's os/exec/exec.go, especially lookExtensions(), then reproduce the bash-on-PATH configuration; done means a Windows agent can resolve the executable from PATH without requiring an absolute path and the regression is covered by an appropriate test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ci-cd, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100