Web request concurrency: gunicorn runs 8 sync workers — evaluate gthread (and why not async Django)

🤖 Written by Claude

## Summary

Gunicorn runs 8 **sync** workers, so the site handles at most 8 concurrent requests. A worker blocked on a slow Postgres query (variant/classification grids, streaming exports) is capacity sitting idle burning no CPU. Switching to `gthread` workers would give ~32 concurrent requests for roughly the same memory, with no application code change.

This issue also records why **async Django is not the answer**, so the question doesn't get re-opened later.

## Why not async Django

In our installed Django 6.1, every async ORM method is a thread-pool wrapper, not a native async driver:

```
django/db/models/query.py:657: async def acount(self): return await sync_to_async(self.count)()
django/db/models/query.py:695: async def aget(self, *a, **k): return await sync_to_async(self.get)(*a, **k)
```

`django/db/backends/postgresql/` contains no `AsyncConnection` usage — psycopg3 is installed but driven synchronously. So `await Variant.objects.aget(...)` hands the query to a worker thread and waits; it does not yield the event loop. For a request path that is overwhelmingly Postgres-bound, async buys nothing the thread pool doesn't already give us.

Cost side: ~206k lines across 1343 files with zero existing `async def`. Async is viral — `GuardianPermissionsMixin.filter_for_user()`, the jqGrid/DataTables base classes, preview signal handlers, `NotificationBuilder`, and every middleware including `GlobalLoginRequiredMiddleware` would need to change or be wrapped. Mixed sync/async Django is where `SynchronousOnlyOperation` and connection-leak bugs live.

Revisit only if Django ships a native async Postgres backend, or if we add websockets (live analysis progress, collaborative classification editing).

## Proposal

```diff
# config/systemd/gunicorn.service
- --bind 127.0.0.1:8000 -t 3600 -w 8 --pid /run/gunicorn/gunicorn.pid \
+ --bind 127.0.0.1:8000 -t 3600 -w 8 --worker-class gthread --threads 4 \
+ --max-requests 1000 --max-requests-jitter 100 --pid /run/gunicorn/gunicorn.pid \
```

`gthread` is built into gunicorn — no new dependency, unlike gevent/eventlet.

| | today (`-w 8` sync) | `-w 8 --threads 4` |
|---|---|---|
| Concurrent requests | 8 | 32 |
| OS processes | 8 | 8 |
| RSS | ~345 MB × 8 ≈ 2.7 GB (measured) | ~same, threads share the heap |
| Blocked on DB | worker idle, capacity lost | other threads keep working |
| Keep-alive / slow clients | consume a whole worker | held by the accept loop |

Reaching 32-way concurrency with sync workers would need `-w 32` ≈ 11 GB RSS, against 15 GB total shared with Postgres and 20 Celery processes.

### Why threads help despite the GIL

Django is on psycopg3 (`is_psycopg3: True`), whose binary implementation releases the GIL while waiting on the server. While one thread waits on a slow grid query, others run template rendering, permission filtering and serialisation. This is **latency hiding, not more CPU** — at 4 cores, `-w 8` is already CPU-oversubscribed.

## Checks before rolling out

**Database connections** — the main risk. `CONN_MAX_AGE: 60` (`default_settings.py:102`) plus thread-local connections means every thread holds its own:

```
today: 8 gunicorn + 20 celery = ~28
with threads: 8 × 4 = 32 + 20 celery = ~52
```

Postgres defaults to `max_connections = 100`. 52 fits, but headroom shrinks and each idle backend costs ~5–10 MB. Verify per deployment:

```bash
sudo -u postgres psql -tAc "show max_connections"
sudo -u postgres psql -c "select state, count(*) from pg_stat_activity group by state"
```

Past ~75 we'd want PgBouncer (transaction mode) or psycopg3 pooling — `psycopg-pool==3.3.1` is already in `requirements.txt`.

**Thread safety** — audited, looks clean:
- No module-level mutable globals in `library`, `snpdb`, `genes`, `annotation`, `classification`, `analysis`
- 3 `global` statements total; 2 are Celery prefork init, off the web path
- The one on the request path (`analysis/views/views_json.py:116`, `NODE_TYPES_HASH`) is an idempotent lazy cache — racing threads compute equivalent dicts, last write wins. Benign.
- 17 `lru_cache`/`@cache` uses, all thread-safe
- No `os.chdir`, no `signal.signal`
- **Still to check:** C-extension singletons, specifically the HGVS converter and SeqRepo objects in `genes/hgvs/`. Per-call or per-request construction is fine; a shared module-level instance needs a look.

**Timeout blast radius** — `-t 3600` guards the worker heartbeat. Under sync workers a recycled worker kills 1 request; under gthread it kills all 4 in-flight on that worker. Mostly theoretical at 3600s, but relevant if we ever tighten the timeout.

## Measurement plan

Roll to one deployment at `--threads 4`, observe for a week:

1. `pg_stat_activity` count settles ~52 and does not climb
2. p95 latency on heavy grid pages — this is what should improve
3. Load average vs 4 cores — if already CPU-saturated, threads surface it as queueing rather than fixing it

Outcomes:
- p95 improves, connections comfortable → consider `--threads 8`
- p95 flat → concurrency was never the bottleneck; a useful negative result pointing at query optimisation
- p95 worse → CPU-bound; the fix is more cores or faster queries

Rollback is deleting the flags and restarting. No code change, no migration.

## Unrelated but adjacent

`config/systemd/gunicorn.service` also runs `--log-level debug`, which writes a lot and costs time on the request path. Worth dropping while we're in that file.

Start with config/systemd/gunicorn.service and default_settings.py:102, then review the thread-safety notes including analysis/views/views_json.py:116 and the shared objects in genes/hgvs/. Check PostgreSQL connection counts with the provided pg_stat_activity commands, deploy --threads 4 to one environment, and observe p95 latency and load for a week. Done means the connection count remains safe and the measured outcome supports rollout, tuning, or rollback.

Written by the indexing model from the issue text.