ooni / ooni/devops

investigate why nginx stops connecting to upstreams

Open
#449 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
HCL
Stars
4
Forks
15
Avg merge
1h 50m
Merged PRs (30d)
4

Description

We noticed that incoming measurements slowed and clients were seeing 400 errors when submitting reports.
I didn't see any issues on the ECS services, but backend-fsn nginx logs showed that it was unable to reach the api endpoint.
However, requests made from backend-fsn had no issues (using curl).

I reloaded nginx and resolved the issue. What I don't know is why this occurred.

Jul 06 08:29:46 backend-fsn.ooni.org nginx[1046013]: backend-fsn.ooni.org ooniapi: xxx 0.0.0.0 - [06/Jul/2026:08:29:46 +0000] "POST /report HTTP/2.0" 502 snt:157 rt:0.000 uprt:0.000 "-" "miniooni/3.24.0 ooniprobe-engine/3.24.0"
Jul 06 08:29:46 backend-fsn.ooni.org nginx[1046013]: backend-fsn.ooni.org ooniapi: xxx 0.0.0.0 MISS [06/Jul/2026:08:29:46 +0000] "GET /api/v1/test-helpers HTTP/2.0" 200 snt:249 rt:0.002 uprt:0.004 "-" "ooniprobe-android-unattended/6.0.2 ooniprobe-engine/3.29.0"
Jul 06 08:29:46 backend-fsn.ooni.org nginx[1046013]: backend-fsn.ooni.org ooniapi: 2026/07/06 08:29:46 [error] 1046013#1046013: *152004051 no live upstreams while connecting to upstream, client: xxx, server: _, request: "POST /report HTTP/2.0", upstream: "https://ooniprobe.prod.ooni.io/report", host: "api.ooni.io"```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the backend-fsn nginx logs and the reported ooniprobe.prod.ooni.io upstream, then compare the nginx upstream configuration with ECS service health and direct curl results. The work is done when the cause of the no-live-upstreams errors is identified and a reproducible explanation or operational fix is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, nginx
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.