investigate why nginx stops connecting to upstreams
Nobody has claimed this yet.
- Dominant language
- HCL
- Stars
- 4
- Forks
- 15
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 4
Description
We noticed that incoming measurements slowed and clients were seeing 400 errors when submitting reports.
I didn't see any issues on the ECS services, but backend-fsn nginx logs showed that it was unable to reach the api endpoint.
However, requests made from backend-fsn had no issues (using curl).
I reloaded nginx and resolved the issue. What I don't know is why this occurred.
Jul 06 08:29:46 backend-fsn.ooni.org nginx[1046013]: backend-fsn.ooni.org ooniapi: xxx 0.0.0.0 - [06/Jul/2026:08:29:46 +0000] "POST /report HTTP/2.0" 502 snt:157 rt:0.000 uprt:0.000 "-" "miniooni/3.24.0 ooniprobe-engine/3.24.0"
Jul 06 08:29:46 backend-fsn.ooni.org nginx[1046013]: backend-fsn.ooni.org ooniapi: xxx 0.0.0.0 MISS [06/Jul/2026:08:29:46 +0000] "GET /api/v1/test-helpers HTTP/2.0" 200 snt:249 rt:0.002 uprt:0.004 "-" "ooniprobe-android-unattended/6.0.2 ooniprobe-engine/3.29.0"
Jul 06 08:29:46 backend-fsn.ooni.org nginx[1046013]: backend-fsn.ooni.org ooniapi: 2026/07/06 08:29:46 [error] 1046013#1046013: *152004051 no live upstreams while connecting to upstream, client: xxx, server: _, request: "POST /report HTTP/2.0", upstream: "https://ooniprobe.prod.ooni.io/report", host: "api.ooni.io"```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the backend-fsn nginx logs and the reported ooniprobe.prod.ooni.io upstream, then compare the nginx upstream configuration with ECS service health and direct curl results. The work is done when the cause of the no-live-upstreams errors is identified and a reproducible explanation or operational fix is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, nginx
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100