kubernetes-sigs / kubernetes-sigs/controller-runtime
level=ERROR msg="error received after stop sequence was engaged" err="leader election lost"
- Dominant language
- Go
- Stars
- 3k
- Forks
- 1.3k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 14
Description
When running a controller manager with `LeaderElection` enabled, shutting down the manager will obviously release the leader lock. However, this seems to still trigger a “leader election lost” error:
```
level=ERROR msg="error received after stop sequence was engaged" err="leader election lost"
```
On its own, this is just a nitpick, but the problem is in combination with using the [t.Output() writer](https://pkg.go.dev/testing#T.Output) in your test functions. Tests are not allowed to keep writing to it after the test has finished. You will either get a panic, or [might even get a data race](https://github.com/golang/go/issues/67701) detected if you run the test with the race detector.
The problem here is that the manager keeps doing something even after its `Start` method has already returned. The log is a symptom of that, but IMO the manager should be completely gone once `Start` returns.
Contributor guide
Research direction
Read the manager Start lifecycle and LeaderElection shutdown path first; reproduce with a controller manager using LeaderElection and a testing.T Output writer. Done means Start does not leave work running or logging after it returns, and shutdown no longer reports leader election lost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100