RPC test server setup panic
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
While doing some backports on nomad Enterprise 1.7.x, I bumped into the following panic for the leadership transfer tests. It looks like the problem is actually in the rpc test server setup though, where the test server isn't always ready at the time we try to dereference its config. I've seen a variant error where the bootstrap token has not replicated to other servers as well, leading to confusing and intermittent auth errors when transferring. This is a test-only issue and doesn't impact production code, but we should get it fixed because it's going to cause a lot of CI re-runs. I haven't seen this on 1.8.x runs, but that could just be good/bad luck with timing.
```
=== RUN TestOperator_TransferLeadershipToServerID_ACL
operator_endpoint_test.go:434: bootstrap token: {24e625d0-6a92-573a-4b4e-74669d3a4b4b 64bcb3a1-8be4-f0a0-650d-f39bb105e12d management 3af9630d-3743-fdb6-e54c-f261cefc92d8 management [] [] true [] 2024-06-14 19:21:55.475905507 +0000 UTC 0s 1 1}
--- FAIL: TestOperator_TransferLeadershipToServerID_ACL (0.75s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x23f8bcf]
goroutine 150195 [running]:
testing.tRunner.func1.2({0x2752940, 0x474ab50})
/opt/hostedtoolcache/go/1.22.4/x64/src/testing/testing.go:1631 +0x24a
testing.tRunner.func1()
/opt/hostedtoolcache/go/1.22.4/x64/src/testing/testing.go:1634 +0x377
panic({0x2752940?, 0x474ab50?})
/opt/hostedtoolcache/go/1.22.4/x64/src/runtime/panic.go:770 +0x132
github.com/hashicorp/nomad/nomad.rpcClient({0x335e610, 0xc009ac9380}, 0x0)
/home/runner/work/nomad-enterprise/nomad-enterprise/nomad/rpc_test.go:46 +0x2f
github.com/hashicorp/nomad/nomad.TestOperator_TransferLeadershipToServerID_ACL(0xc009ac9380)
/home/runner/work/nomad-enterprise/nomad-enterprise/nomad/operator_endpoint_test.go:530 +0x133
testing.tRunner(0xc009ac9380, 0x2c61918)
/opt/hostedtoolcache/go/1.22.4/x64/src/testing/testing.go:1689 +0xfb
created by testing.(*T).Run in goroutine 1
/opt/hostedtoolcache/go/1.22.4/x64/src/testing/testing.go:1742 +0x390
FAIL nomad.TestOperator_TransferLeadershipToServerID_ACL (0.75s)
FAIL nomad
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with nomad/rpc_test.go at rpcClient and nomad/operator_endpoint_test.go around TestOperator_TransferLeadershipToServerID_ACL and the reported lines. Reproduce the leadership-transfer test to observe the server setup timing; done means the test no longer panics or fails intermittently when the server config or bootstrap token is not yet ready.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100