decentraland / decentraland/actions

Deploy workflows report success before the deployment completes

Open
#20 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Problem

Deploy workflows report **success** as soon as the deployment record is created, not when the deployment finishes. `deploy-service` creates the GitHub Deployment, posts `queued`, and returns. The rollout itself happens out of band: `webhooks-receiver` picks up the deployment webhook and triggers the Pulumi pipeline in `ops/services-pipeline`. Nothing in the workflow ever looks at the outcome.

`Trigger deployment` is the last step in the deployment job of all three workflows:

- `.github/workflows/build-quay-main.yml`
- `.github/workflows/deploy-latest.yml`
- `.github/workflows/manual-deploy-service.yml`

## Evidence

**1. A successful deploy, timed.** `squid-management-server` → dev on 2026-08-14:

```
workflow job "Deploy to: dev" success at 15:20:46Z
deployment 5908704802 success at 15:31:29Z
```

The run went green **10m43s before the deployment finished**. Everything that can actually fail — Pulumi update, task-definition replacement, ECS rollout, health checks — happens after the green check.

**2. A run that was green while the deployment could never complete.** The `manual-deploy-service.yml` dispatch on the same day:

```
run 31802348624 conclusion: success
deployment 5906488060 never reached a terminal state
```

The status callback 404'd, the record sat `queued` indefinitely, and Actions reported success throughout.

## Why this matters

This is the failure mode behind the 2026-08-07 `realm-provider` incident. A malformed image reference was deployed to prd; the Actions run reported **success**, and the deployment reached `failure` 18 minutes later:

```
run 31177484161 conclusion = success
deployment 5794322360 12:16:25 queued -> 12:18:37 in_progress -> 12:34:19 FAILURE
```

29 failed ECS tasks and a Prometheus DOWN alert were the only signal. The receiver **does** post terminal states (`success` / `failure` / `inactive`), so the outcome is observable — nothing is polling it.

## Proposed fix

`deploy-service` already exposes a `deployment-ids` output (JSON array, one id per environment). That makes the exact-id version possible here, with no correlation heuristics:

```yaml
- name: Wait for deployment outcome
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
IDS: ${{ steps.deploy.outputs.deployment-ids }}
run: |
set -euo pipefail
for id in $(printf '%s' "$IDS" | jq -r '.[]'); do
for _ in $(seq 1 60); do # 60 x 30s = 30 min
state=$(gh api "repos/${GITHUB_REPOSITORY}/deployments/${id}/statuses" --jq '.[0].state')
case "$state" in
success) echo "deployment ${id} succeeded"; break ;;
failure|error) echo "::error::Deployment ${id} ended in state=${state}"; exit 1 ;;
inactive) echo "deployment ${id} superseded by a later deployment"; break ;;
esac
sleep 30
done
done
```

Iterating the array matters: `env` accepts several environments space separated (`explorer-metrics` passes `dev prd`), and `deploy-service` creates one deployment per environment.

## Per-workflow notes

**`manual-deploy-service.yml` needs a decision first.** Its deployments are created on this repo, but the pipeline posts status back to the repo the service definition points to, so the callback always 404s and the record can never reach a terminal state. Adding a wait step there would convert a silently permanent `queued` into a guaranteed 30-minute timeout failure. That is arguably the honest outcome, but it should be a deliberate choice: either fix the repo mismatch, delete the workflow, or accept that it will always fail loudly. Tracked separately.

`build-quay-main.yml` and `deploy-latest.yml` create deployments in the caller's repo, which is the service's own repo, so their callbacks work — confirmed by deployment 5908704802 reaching `success`.

## Caveats

- Observed durations are 3-18 minutes, so 30 minutes is the right ceiling. The job then holds a runner for that long instead of finishing in seconds.
- `inactive` is a **legitimate** terminal state when a later deployment supersedes yours (observed on `realm-provider` dep 5797225668). Do not treat it as failure.
- Deploys that have always been silently half-broken will start showing red. That is the point, but worth announcing before rollout.

## Context

Supersedes decentraland/platform-actions#64, which described the same defect for the `apps-docker-*` workflows. That issue proposed correlating by `sha` + `environment` because the archived `dcl-deploy-action` exposes no deployment id — a workaround that is ambiguous under concurrent deploys of the same commit to the same environment. Since `deploy-service` emits ids directly, the clean version is available here.

Fixing it centrally in this repo also fits the direction of travel: the `platform-actions` workflows are to be repointed at `decentraland/actions` over time, so improvements land once rather than being implemented twice with different mechanics.

Contributor guide

Open the contributing guide

Research direction

Start with the final Trigger deployment steps in .github/workflows/build-quay-main.yml, .github/workflows/deploy-latest.yml, and .github/workflows/manual-deploy-service.yml, then inspect deploy-service's deployment-ids output. Confirm the two caller workflows wait for each deployment's terminal outcome and fail on failure or error; resolve the documented repo-mismatch decision for manual-deploy-service.yml before changing it.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, shell
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.