Audit Cloud Run Concurrency Settings Across All Services

=== Background ===

Performance investigation revealed containerConcurrency/GUNICORN_THREADS mismatches across services causing request queuing during load spikes.

Common Pattern:
containerConcurrency: 10 (Cloud Run accepts 10 concurrent requests)
GUNICORN_THREADS: 8 (But app can only handle 8)
Result: 2+ requests queue, waiting 30-60+ seconds during concurrent load

Root Cause: When all threads are busy with slow operations, excess requests queue in gunicorn until a thread frees up, causing severe latency spikes.

Evidence: Services show normal operation time (5-10s) but end-to-end latency can spike to 50-60s = queue wait + execution time.

=== Investigation Tasks ===

For each service, check:
CPU: [cores]
Memory: [GB]
containerConcurrency: [current value]
GUNICORN_THREADS: [current value]
GUNICORN_PROCESSES: [current value] - this should be 1 for CloudRun (as per gcp recommendation)
Workload type: [CPU-bound / I/O-bound / mixed]

3. Apply these rules:

Workload Type Formula Example (4 CPU)
CPU-bound (compute) threads = CPU cores 4 threads
I/O-bound (proxy, DB) threads = 3-5 × CPU cores 12-20 threads
ALWAYS: containerConcurrency = threads

4. Memory considerations:
- Each thread holds request/response data in memory
- Conservative estimate: 50-100MB per thread for JSON APIs
- Verify: memory_limit / threads > 100MB

=== Recommended Fixes ===

For each service with mismatch:

Increase threads to match concurrency
- Update GUNICORN_THREADS to match containerConcurrency
- Verify sufficient CPU and memory resources

Update vault locations: op://[service]/[env]/[config-path]/GUNICORN_THREADS

=== Related Issues ===

- Performance: Occasional 30-60+ second queries in production resulting in latency alerts

Start by locating the Cloud Run configuration for each service and its corresponding vault entry at the stated GUNICORN_THREADS path. Record CPU, memory, concurrency, thread and process settings, classify each workload, and verify that every mismatch is corrected with sufficient memory.

Written by the indexing model from the issue text.