PodRestartErrors are too noisy/sticky
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 10.1k
- Forks
- 413
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 6
Description
A user saw yellow "has alert" indications on a number of pods and didn't understand why/thought there was something serious wrong--turns out they were just PodRestartError notifications. But sometimes pods just restart and it's not a big deal/is even expected.
Given that PodRestarts are sometimes not a big deal, we should consider surfacing this error in way that makes this clearer to users, otherwise they spend a lot of time trying to debug/hunting for errors.
@theothertomelliott suggests maybe PodRestartErrors expire a certain amount of time after they were first reported.
There IS a manual "dismiss" button, but it's a bit of extra effort/requires devs to understand what's going on enough to know they can dismiss these alerts. Tom notes:
I'm less enthusiastic about manual dismissal of alerts, just because we'll have a lot of different engineers using these Tiltfiles and they won't all be aware of this behavior, so could spend a lot of time trying to debug
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the PodRestartError notification and its manual dismiss behavior. Review the discussion about expiration and define completion as presenting routine pod restarts more clearly, potentially with alerts expiring after they are reported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100