`clients_requesting_changes` gauge not decrementing when clients disconnect in CouchDB 3.5
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ )
## Description
The `couchdb_httpd_clients_requesting_changes` gauge metric is not properly decremented when clients disconnect from the `_changes` feed. The metric continues to show the cumulative count of connections rather than the current active connection count.
Interestingly, the `abandoned_streaming_requests` counter IS being incremented when disconnections are detected, indicating that the disconnect detection logic is working, but the main gauge is not being updated.
## Steps to Reproduce
### Setup
1. Start CouchDB 3.5 using Docker:
```bash
docker run -d --name couchdb -p 5984:5984 \
-e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password \
apache/couchdb:3.5
```
2. Create a test database:
```bash
curl -X PUT -u admin:password http://localhost:5984/testdb
```
### Reproduce the issue
3. Check initial stats:
```bash
curl -s -u admin:password "http://localhost:5984/_node/_local/_stats/couchdb/httpd/clients_requesting_changes"
# Expected: {"value":0,"type":"counter","desc":"number of clients for continuous _changes"}
```
4. Start 4 continuous changes feed connections (in separate terminals or background processes):
```bash
# Terminal 1
curl -u admin:password "http://localhost:5984/testdb/_changes?feed=continuous&heartbeat=10000"
# Terminal 2
curl -u admin:password "http://localhost:5984/testdb/_changes?feed=continuous&heartbeat=10000"
# Terminal 3
curl -u admin:password "http://localhost:5984/testdb/_changes?feed=continuous&heartbeat=10000"
# Terminal 4
curl -u admin:password "http://localhost:5984/testdb/_changes?feed=continuous&heartbeat=10000"
```
5. Verify connections are counted:
```bash
curl -s -u admin:password "http://localhost:5984/_node/_local/_stats/couchdb/httpd/clients_requesting_changes"
# Expected: {"value":4,...}
```
6. Stop all 4 curl processes (Ctrl+C on each terminal)
7. Check stats again:
```bash
curl -s -u admin:password "http://localhost:5984/_node/_local/_stats/couchdb/httpd/clients_requesting_changes"
# ACTUAL: {"value":4,...} - still shows 4!
# EXPECTED: {"value":0,...}
```
8. Check abandoned_streaming_requests:
```bash
curl -s -u admin:password "http://localhost:5984/_node/_local/_stats/couchdb/httpd/abandoned_streaming_requests"
# Shows: {"value":4,...} - correctly detected 4 abandonments
```
9. Start 4 new connections again and stop them:
```bash
# Start 4 connections, then stop them with Ctrl+C
```
10. Check stats:
```bash
curl -s -u admin:password "http://localhost:5984/_node/_local/_stats/couchdb/httpd/clients_requesting_changes"
# ACTUAL: {"value":8,...} - doubled!
# EXPECTED: {"value":0,...}
curl -s -u admin:password "http://localhost:5984/_node/_local/_stats/couchdb/httpd/abandoned_streaming_requests"
# Shows: {"value":8,...} - correctly shows 8 total abandonments
The Prometheus endpoint correctly types this as a gauge, but the value doesn't decrease:
```bash
curl -s -u admin:password "http://localhost:5984/_node/_local/_prometheus" | grep clients_requesting_changes
# TYPE couchdb_httpd_clients_requesting_changes gauge
# couchdb_httpd_clients_requesting_changes 8
```
## Expected Behaviour
The `clients_requesting_changes` metric should:
1. Increment when a client starts a continuous/longpoll `_changes` feed
2. Decrement when the client disconnects (either normally or via TCP close)
3. Always reflect the **current** number of active changes feed connections
## Actual Behavior
- The metric increments correctly when connections start
- The metric does **NOT** decrement when connections end
- The `abandoned_streaming_requests` counter correctly tracks disconnections
- This causes the gauge to act like a monotonic counter, accumulating over time
## Your Environment
* CouchDB version used: 3.5.1
* Browser name and version: Brave 1.84.139
* Operating system and version: macOS (Darwin 24.6.0) / Docker
* **Installation method**: Docker (apache/couchdb:3.5 image)
## Additional Context
This issue was discovered while upgrading from CouchDB 3.3.1 to 3.5.1. The metric worked correctly in 3.3.1.
The issue affects monitoring and alerting systems that rely on this metric to track the number of active changes feed consumers.
Contributor guide
Research direction
Start at the CouchDB _changes HTTP feed entry point and the stats or Prometheus output for couchdb_httpd_clients_requesting_changes; compare the increment path with disconnect handling, using abandoned_streaming_requests as evidence. Reproduce with the Docker and curl commands, then verify that active connections return to zero after clients stop while the abandonment counter remains cumulative.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang
- Domain
- backend, databases, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100