tilt-dev / tilt-dev/tilt

PodRestartErrors are too noisy/sticky

Open
#3,483 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
10.1k
Forks
413
Avg merge
1d 10h
Merged PRs (30d)
6

Description

A user saw yellow "has alert" indications on a number of pods and didn't understand why/thought there was something serious wrong--turns out they were just PodRestartError notifications. But sometimes pods just restart and it's not a big deal/is even expected.

Given that PodRestarts are sometimes not a big deal, we should consider surfacing this error in way that makes this clearer to users, otherwise they spend a lot of time trying to debug/hunting for errors.

@theothertomelliott suggests maybe PodRestartErrors expire a certain amount of time after they were first reported.

There IS a manual "dismiss" button, but it's a bit of extra effort/requires devs to understand what's going on enough to know they can dismiss these alerts. Tom notes:

I'm less enthusiastic about manual dismissal of alerts, just because we'll have a lot of different engineers using these Tiltfiles and they won't all be aware of this behavior, so could spend a lot of time trying to debug

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the PodRestartError notification and its manual dismiss behavior. Review the discussion about expiration and define completion as presenting routine pod restarts more clearly, potentially with alerts expiring after they are reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.