[RCA incident] Multiple Application Unreachable

**RCA Discussion Thread**
https://juspay.slack.com/archives/C04QY9BFM7Y/p1784268633489389?thread_ts=1784268601.064029&cid=C04QY9BFM7Y

**Detection Method**
Pingdom Alerts

**Incident Triggered By**
[AWS outage](https://health.aws.amazon.com/health/status?eventID=arn:aws:health:global::event/CLOUDFRONT/AWS_CLOUDFRONT_OPERATIONAL_ISSUE/AWS_CLOUDFRONT_OPERATIONAL_ISSUE_EDF4C_83808542A4E)

**Executive Summary**

AWS VPC origins started returning 504.
Affected domains
| Service | Sandbox | Production |
|--------|--------|--------|
| superposition.juspay.io | DOWN | DOWN |
| aarokya.juspay.io | UP | DOWN |
| airborne.juspay.in | DOWN | DOWN |

**Timelines - 16th July 2026**

| Timestamp | what happened |
|--------|--------|
| 1:15 | AWS VPC origin outage event start time |
| 1:25 | Pingdom alerts raised for aarokya |
| 1:29 | Pingdom alerts raised for superposition |
| 1:40 | Meet was started |
| 1:46 | Raise a support ticket with AWS support |
| 1:54 | Started a chime call with AWS support |
| ?? | Curls to envoy |
| 2:19 | Created NLB to proxy to ALB |
| 2:27 | Switch traffic to NLB for superposition. Pingdom up alert fired |
| 2:43 | Switch traffic to NLB for aarokya. Pingdom up alert fired |

**IMPACT**
All applications were not reachable via cloudfront VPC origins.
Aarokya
1. Servers were unreachable
2. 1 user mandate execution failed

Superposition
1. All superposition client which rebooted during the window would fail. Actual numbers are unknown.
2. Releases were not possible in superposition
3. In memory caches in CAC clients protected all users using the CAC client

Airborne
1. Releases were not possible
2. Fresh users are stuck on older versions
3. Cloudfront caches protected most of airborne traffic.

There was 1 and half hour downtime across all applications

**EVIDENCE**

Superposition Cloudfront

Airborne Cloudfront (Low rates since it is highly protected by cache)

Superposition + Airborne ALB

Aarokya Cloudfront

Aarokya ALB

Q. Why did we take 80 mins to recover?
A. Because we were not sure of the diagnosis

Q. Why were we not sure of the diagnosis?
A. Because we did not follow a systematic triage. And we did not set up a standard monitoring dashboard

**COEs**