NASA-IMPACT / NASA-IMPACT/veda-deploy

[stretch] Alert when a veda-backend instance hasn't been deployed in a configurable window

Open
#208 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Support Engineering
Dominant language
Python
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Summary

We currently have no way of knowing when an instance is running a stale version of veda-backend unless someone manually checks. A lightweight alerting mechanism would help us catch instances that fall behind, particularly after breaking upstream changes (like the recent stac-fastapi 6.3 incompatibility) see more here : https://github.com/NASA-IMPACT/veda-backend/issues/623

When a breaking change lands in a dependency or we cut a release with a critical fix, there's no signal that an instance is still running an older version. This has led to situations where we discover incompatibilities at deploy time rather than proactively.

Possible approaches
  • GitHub Actions scheduled workflow that checks deployed instances against the latest release tag and opens an issue or sends a Slack notification if the gap exceeds a threshold?
  • CloudWatch/EventBridge rule that fires if no deployment event has been recorded for an environment within a configurable window
  • Could we maybe use Grafana and add a widget (not sure if that's the correct term) showing last deploy date per instance (lower effort, no alerting, but makes staleness visible)
Acceptance Criteria
  • Define what "stale" means (time-based, version-based, or both)
  • At least one instance type (starting with veda-backend) is covered
  • Notification is sent to a team-visible channel (Slack, GitHub issue, or email)
  • Threshold is configurable

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No specific file or test is named. Start by reviewing the repository's deployment workflows and available deployment events, then compare the scheduled GitHub Actions, CloudWatch/EventBridge, and Grafana approaches. Done means stale instances are defined, at least one veda-backend instance type is covered, the threshold is configurable, and a team-visible notification is sent.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions, grafana
Domain
devops, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.