ProjectTech4DevAI / ProjectTech4DevAI/kaapi-backend

CI/CD: Improve deployment security

Open
#1,156 2 comments 0 reactions 1 assignee View on GitHub

@Ayush8923 is already working on this.

Since Aug 25, 2026.

  • #1158 by @Ayush8923 — merged
infrastructure security
Dominant language
Python
Stars
18
Forks
10
Avg merge
2d 20h
Merged PRs (30d)
14

Description

Is your feature request related to a problem?
Our CI/CD pipelines have vulnerabilities that could lead to deployment incidents, similar to what occurred with the Glific team. Addressing these gaps is crucial to ensure deployment success and minimize failures.

Describe the solution you'd like

  • Ensure that Staging deploys wait for a successful CI run by updating deploy-staging.yml with workflow_run and adding branch protection on main.
  • Integrate a Discord webhook step to notify about staging and production deploy failures.
  • Implement health checks for deployments by enabling the ECS deployment circuit breaker and polling the rolloutState.
  • Add Docker image builds in CI for every PR to catch issues earlier.
  • Set up ECS staging rehearsals for every merge to main by scaling the staging ECS service and verifying it, with notifications on failures.
Original issue

Our CI CD pipelines can be hardened. There are gaps that need fixing, which will prevent from us from having similar deployment incidents like what Glific team shared recently.
More details here

1. Deploys wait for CI to be green

Staging deploy currently races CI instead of waiting for it. Change deploy-staging.yml to trigger only after a successful CI run (workflow_run), add branch protection on main with required checks, and make create-release.yml confirm the tagged commit passed CI before building.

2. Discord notifications on deploy failures

Add a Discord webhook step (if: failure()) to the staging and production deploy workflows. Today a red deploy on main is only visible if someone opens the Actions tab — and Sentry can't catch failures that happen before the app boots.

3. Verify deployments actually became healthy
  • Enable the ECS deployment circuit breaker on our services.
  • After update-service, poll the deployment's rolloutState until COMPLETED (pass) or FAILED / timeout (fail). A plain curl isn't enough — the old task can answer with 200 while the new one is failing.
  • Bake the git SHA into the image and return it from the health endpoint, so we can confirm the new code is the one responding.
  • On the EC2 staging box, wait for the container healthcheck after docker compose up -d instead of reporting success when containers start.
4. Build the Docker image in CI

on every PR CI never builds the Dockerfile or validates the compose files, so breakage there is only discovered during a deploy. Add docker build ./backend and docker compose config checks to CI.

5. ECS staging rehearsal on every merge to main

Staging runs on EC2 (to save costs), so the production ECS deploy path is never tested before a release. After the EC2 staging deploy completes (serial, not parallel — the EC2 path owns migrations), scale the staging ECS service from 0 to 1 with the new image, verify with the rolloutState poll, then scale back to 0 (if: always()). One-time prep: remove autoscaling on the staging service and enable its circuit breaker. A failed rehearsal pings Discord but does not roll back EC2 staging.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.