hashicorp / hashicorp/nomad

RPC test server setup panic

Open
#23,337 0 comments 0 reactions 0 assignees View on GitHub
hcc/jira theme/flaky-tests type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

While doing some backports on nomad Enterprise 1.7.x, I bumped into the following panic for the leadership transfer tests. It looks like the problem is actually in the rpc test server setup though, where the test server isn't always ready at the time we try to dereference its config. I've seen a variant error where the bootstrap token has not replicated to other servers as well, leading to confusing and intermittent auth errors when transferring. This is a test-only issue and doesn't impact production code, but we should get it fixed because it's going to cause a lot of CI re-runs. I haven't seen this on 1.8.x runs, but that could just be good/bad luck with timing.

```
=== RUN TestOperator_TransferLeadershipToServerID_ACL
operator_endpoint_test.go:434: bootstrap token: {24e625d0-6a92-573a-4b4e-74669d3a4b4b 64bcb3a1-8be4-f0a0-650d-f39bb105e12d management 3af9630d-3743-fdb6-e54c-f261cefc92d8 management [] [] true [] 2024-06-14 19:21:55.475905507 +0000 UTC 0s 1 1}
--- FAIL: TestOperator_TransferLeadershipToServerID_ACL (0.75s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x23f8bcf]

goroutine 150195 [running]:
testing.tRunner.func1.2({0x2752940, 0x474ab50})
/opt/hostedtoolcache/go/1.22.4/x64/src/testing/testing.go:1631 +0x24a
testing.tRunner.func1()
/opt/hostedtoolcache/go/1.22.4/x64/src/testing/testing.go:1634 +0x377
panic({0x2752940?, 0x474ab50?})
/opt/hostedtoolcache/go/1.22.4/x64/src/runtime/panic.go:770 +0x132
github.com/hashicorp/nomad/nomad.rpcClient({0x335e610, 0xc009ac9380}, 0x0)
/home/runner/work/nomad-enterprise/nomad-enterprise/nomad/rpc_test.go:46 +0x2f
github.com/hashicorp/nomad/nomad.TestOperator_TransferLeadershipToServerID_ACL(0xc009ac9380)
/home/runner/work/nomad-enterprise/nomad-enterprise/nomad/operator_endpoint_test.go:530 +0x133
testing.tRunner(0xc009ac9380, 0x2c61918)
/opt/hostedtoolcache/go/1.22.4/x64/src/testing/testing.go:1689 +0xfb
created by testing.(*T).Run in goroutine 1
/opt/hostedtoolcache/go/1.22.4/x64/src/testing/testing.go:1742 +0x390
FAIL nomad.TestOperator_TransferLeadershipToServerID_ACL (0.75s)
FAIL nomad
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with nomad/rpc_test.go at rpcClient and nomad/operator_endpoint_test.go around TestOperator_TransferLeadershipToServerID_ACL and the reported lines. Reproduce the leadership-transfer test to observe the server setup timing; done means the test no longer panics or fails intermittently when the server config or bootstrap token is not yet ready.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.