ProjectTech4DevAI / ProjectTech4DevAI/kaapi-backend

CI/CD: Improve deployment security

Aperta
#1,156 2 commenti 0 reazioni 1 assegnatario Vedi su GitHub

@Ayush8923 ci sta già lavorando.

Dal 25/8/2026.

  • #1158 di @Ayush8923 — integrata
infrastructure security
Lingua principale
Python
Stelle
18
Fork
10
Merge medio
2g 20h
PR unite (30g)
14

Descrizione

Is your feature request related to a problem?
Our CI/CD pipelines have vulnerabilities that could lead to deployment incidents, similar to what occurred with the Glific team. Addressing these gaps is crucial to ensure deployment success and minimize failures.

Describe the solution you'd like

  • Ensure that Staging deploys wait for a successful CI run by updating deploy-staging.yml with workflow_run and adding branch protection on main.
  • Integrate a Discord webhook step to notify about staging and production deploy failures.
  • Implement health checks for deployments by enabling the ECS deployment circuit breaker and polling the rolloutState.
  • Add Docker image builds in CI for every PR to catch issues earlier.
  • Set up ECS staging rehearsals for every merge to main by scaling the staging ECS service and verifying it, with notifications on failures.
Original issue

Our CI CD pipelines can be hardened. There are gaps that need fixing, which will prevent from us from having similar deployment incidents like what Glific team shared recently.
More details here

1. Deploys wait for CI to be green

Staging deploy currently races CI instead of waiting for it. Change deploy-staging.yml to trigger only after a successful CI run (workflow_run), add branch protection on main with required checks, and make create-release.yml confirm the tagged commit passed CI before building.

2. Discord notifications on deploy failures

Add a Discord webhook step (if: failure()) to the staging and production deploy workflows. Today a red deploy on main is only visible if someone opens the Actions tab — and Sentry can't catch failures that happen before the app boots.

3. Verify deployments actually became healthy
  • Enable the ECS deployment circuit breaker on our services.
  • After update-service, poll the deployment's rolloutState until COMPLETED (pass) or FAILED / timeout (fail). A plain curl isn't enough — the old task can answer with 200 while the new one is failing.
  • Bake the git SHA into the image and return it from the health endpoint, so we can confirm the new code is the one responding.
  • On the EC2 staging box, wait for the container healthcheck after docker compose up -d instead of reporting success when containers start.
4. Build the Docker image in CI

on every PR CI never builds the Dockerfile or validates the compose files, so breakage there is only discovered during a deploy. Add docker build ./backend and docker compose config checks to CI.

5. ECS staging rehearsal on every merge to main

Staging runs on EC2 (to save costs), so the production ECS deploy path is never tested before a release. After the EC2 staging deploy completes (serial, not parallel — the EC2 path owns migrations), scale the staging ECS service from 0 to 1 with the new image, verify with the rolloutState poll, then scale back to 0 (if: always()). One-time prep: remove autoscaling on the staging service and enable its circuit breaker. A failed rehearsal pings Discord but does not roll back EC2 staging.

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.