ethereum-optimism / ethereum-optimism/optimism

circleci: Add alerts if develop-tagged images are out of date due to CI workflows not running

Open
#22,428 0 comments 0 reactions 0 assignees View on GitHub
C-observability M-ci T-proofs
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

In #22421, it was noticed that the `kona-publish-prestates` job has not been running for an extended amount of time. This should be something we could have detected earlier with an external slack/grafana alert, checking if the latest develop images that we expect to be published by CI workflows on develop pushes, are out of date or not.

We should make sure to add freshness alerts for all develop-tracking images published by CI workflows, and alert if the latest pushed image is falling behind the current develop head commit.

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the CircleCI workflows that publish develop-tracking images and the existing Grafana alerting setup. Trace how the current develop head commit and latest pushed image can be compared. Done means freshness alerts cover all develop-tracking images and report when an image falls behind develop.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana
Domain
ci-cd, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.