agent panic on windows during bootstrap
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 378
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 74
Description
We recently saw two builds [1](https://buildkite.com/ray-project/ray-builders-branch/builds/5486#6f2799d7-2c22-4105-b981-b42ccf5bdb4b) [2](https://buildkite.com/ray-project/ray-builders-branch/builds/5486#6f2799d7-2c22-4105-b981-b42ccf5bdb4b) failed to execute with this shown in the log:
```
fatal error: unexpected signal during runtime execution
[signal 0xc0000005 code=0x1 addr=0x4fe695ac7ff pc=0x32fc89]
goroutine 8 [running]:
runtime.throw(0xcdfec9, 0x2a)
/usr/local/go/src/runtime/panic.go:1117 +0x79 fp=0xc000053dc8 sp=0xc000053d98 pc=0x31c159
runtime.sigpanic()
/usr/local/go/src/runtime/signal_windows.go:233 +0x317 fp=0xc000053e20 sp=0xc000053dc8 pc=0x332157
runtime.(*waitq).enqueue(...)
/usr/local/go/src/runtime/chan.go:784
runtime.selectgo(0xc000053fa0, 0xc000053f98, 0x0, 0x0, 0x2, 0x1, 0x0, 0x0)
/usr/local/go/src/runtime/select.go:316 +0xdc9 fp=0xc000053f58 sp=0xc000053e20 pc=0x32fc89
github.com/buildkite/agent/v3/bootstrap.(*Bootstrap).Run.func1(0xc0002e1810, 0xc0002cb900)
/work/bootstrap/bootstrap.go:93 +0x94 fp=0xc000053fd0 sp=0xc000053f58 pc=0xaca814
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000053fd8 sp=0xc000053fd0 pc=0x354ae1
created by github.com/buildkite/agent/v3/bootstrap.(*Bootstrap).Run
/work/bootstrap/bootstrap.go:92 +0xfa
goroutine 1 [runnable]:
syscall.Environ(0x0, 0x0, 0x0)
/usr/local/go/src/syscall/env_windows.go:83 +0x185
os.Environ(...)
/usr/local/go/src/os/env.go:140
github.com/buildkite/agent/v3/bootstrap.(*Bootstrap).setUp(0xc0002cb900, 0xe45420, 0xc00029d740, 0x0, 0x0)
/work/bootstrap/bootstrap.go:591 +0xff
github.com/buildkite/agent/v3/bootstrap.(*Bootstrap).Run(0xc0002cb900, 0xe45378, 0xc000299700, 0x0)
/work/bootstrap/bootstrap.go:119 +0x214
github.com/buildkite/agent/v3/clicommand.glob..func8(0xc0001354a0)
/work/clicommand/bootstrap.go:448 +0x9ca
github.com/urfave/cli.HandleAction(0xb68e00, 0xcff558, 0xc0001354a0, 0xc0001354a0, 0x0)
/go/pkg/mod/github.com/urfave/cli@v1.22.4/app.go:528 +0x62
github.com/urfave/cli.Command.Run(0xcb6a2e, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0xccb2a0, 0x1b, 0x0, ...)
/go/pkg/mod/github.com/urfave/cli@v1.22.4/command.go:174 +0x579
github.com/urfave/cli.(*App).Run(0xc0002a41c0, 0xc0000343e0, 0x2, 0x2, 0x0, 0x0)
/go/pkg/mod/github.com/urfave/cli@v1.22.4/app.go:279 +0x808
main.main()
/work/main.go:123 +0x6c5
goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:168 +0xaf
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x29
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:151 +0x4b
goroutine 7 [chan receive]:
github.com/buildkite/agent/v3/clicommand.glob..func8.1(0xc00005b860, 0xc0002bb9b0, 0xc0002cb900, 0xc0002bb998, 0xc0002e1800)
/work/clicommand/bootstrap.go:432 +0x55
created by github.com/buildkite/agent/v3/clicommand.glob..func8
/work/clicommand/bootstrap.go:431 +0x99f
```
Not sure how diagnosable it is but it just started happen in the past 100 commits (so far it happened 2/100) times. Feel free to close if this is not actionable/hard to reproduce. Thank you
```
export BUILDKITE_STACK_VERSION="v5.2.0"
```
Contributor guide
Research direction
Start with the Windows stack trace and the bootstrap paths in bootstrap/bootstrap.go, especially setUp and Run, then inspect clicommand/bootstrap.go around the reported lines. Compare the recent 100 commits and reproduce the bootstrap command if possible; done means identifying and fixing the intermittent Windows panic with coverage for the failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ci-cd, cli, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100