kubernetes-sigs / kubernetes-sigs/controller-runtime

level=ERROR msg="error received after stop sequence was engaged" err="leader election lost"

Open
#3,535 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3k
Forks
1.3k
Avg merge
2d 5h
Merged PRs (30d)
14

Description

When running a controller manager with `LeaderElection` enabled, shutting down the manager will obviously release the leader lock. However, this seems to still trigger a “leader election lost” error:

```
level=ERROR msg="error received after stop sequence was engaged" err="leader election lost"
```

On its own, this is just a nitpick, but the problem is in combination with using the [t.Output() writer](https://pkg.go.dev/testing#T.Output) in your test functions. Tests are not allowed to keep writing to it after the test has finished. You will either get a panic, or [might even get a data race](https://github.com/golang/go/issues/67701) detected if you run the test with the race detector.

The problem here is that the manager keeps doing something even after its `Start` method has already returned. The log is a symptom of that, but IMO the manager should be completely gone once `Start` returns.

Contributor guide

Open the contributing guide

Research direction

Read the manager Start lifecycle and LeaderElection shutdown path first; reproduce with a controller manager using LeaderElection and a testing.T Output writer. Done means Start does not leave work running or logging after it returns, and shutdown no longer reports leader election lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.