apache / apache/couchdb

Scheduler-rotated replication jobs are incorrectly closing sockets

Open
#1,417 0 comments 0 reactions 0 assignees View on GitHub
bug replication
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

# Description

This is a bit convoluted, so bear with me.

One deployment of CouchDB in a db-per-user type scenario has many more filtered continuous replication jobs than `max_jobs`, and increasing `max_jobs` to match is impractical (CPU limits). As the scheduler rotates through jobs, we are seeing the number of TIME_WAIT sockets on the nodes drastically increase.

# Expected behaviour

When the replication scheduler rotates off (*i.e.* kills) a continuous replication job to round-robin to another waiting replication job, the socket should be closed on the client side before killing replication, triggering proper socket cleanup.

# Current behaviour

When the replication scheduler rotates off (*i.e.* kills) a continuous replication job to round-robin to another waiting replication job, the socket is *not* closed correctly, leaving a client socket in TIME_WAIT to expire.

# Steps to reproduce

First, set up a test environment. In one window:

```
dev/run -n 1 --with-admin-party-please
```

In another window, set up a short script to monitor the number of TIME_WAIT TCP connections:

```
watch -n 1 "netstat -an | grep TIME_WAIT | grep 15984 | wc -l"
```

Leave off the `| wc -l` if you want to see a full list.

Now, in a *third* window, prep 6 test databases:
```
curl -X PUT localhost:15984/abc
curl -X PUT localhost:15984/one
curl -X PUT localhost:15984/two
curl -X PUT localhost:15984/three
curl -X PUT localhost:15984/four
curl -X PUT localhost:15984/five
```

Create continuous replication documents to replicate from the shared `abc` database to each of the "db-per-user" databases:
```
curl -X PUT localhost:15984/_replicator/one?rev=2-3efcb2be4a2cf5430a56d8919e8fdd79 -d '{"source": "http://localhost:15984/abc", "target": "http://localhost:15984/one", "continuous": true}'
curl -X PUT localhost:15984/_replicator/two -d '{"source": "http://localhost:15984/abc", "target": "http://localhost:15984/two", "continuous": true}'
curl -X PUT localhost:15984/_replicator/three -d '{"source": "http://localhost:15984/abc", "target": "http://localhost:15984/three", "continuous": true}'
curl -X PUT localhost:15984/_replicator/four -d '{"source": "http://localhost:15984/abc", "target": "http://localhost:15984/four", "continuous": true}'
curl -X PUT localhost:15984/_replicator/five -d '{"source": "http://localhost:15984/abc", "target": "http://localhost:15984/five", "continuous": true}'
```

Finally, force the replicator to churn by adjusting the replicator max jobs, interval, and startup jitter to minimal values:
```
curl -X PUT localhost:15984/_node/_local/_config/replicator/interval -d '"1000"'
curl -X PUT localhost:15984/_node/_local/_config/replicator/max_jobs -d '"1"'
curl -X PUT localhost:15984/_node/_local/_config/replicator/startup_jitter -d '"1"'
```

Now, the replicator scheduler will only have a single job running at a time, and will rotate through jobs every second, with 1ms of jitter on starting each job.

Sitting at idle, with 0 documents in the database, this is currently showing a steady state of ~50-60 TIME_WAIT sockets. Looking at the output of `netstat -an | grep 15984`, all of these sockets show port 15984 as the destination. Example:

```
tcp 0 0 127.0.0.1:46031 127.0.0.1:15984 TIME_WAIT
```

All of the TIME_WAIT sockets are client-side; that is, I never see `127.0.0.1:15984` in the source column of the `netstat` output unless I kill CouchDB (obviously)

-----
Some interesting things to try at this point:

* In a separate window, perturb the abc database by adding a single document:
```
curl -X POST -H "Content-Type: application/json" localhost:15984/abc -d '{"field": "1"}'; done
```
* Apply the following tweak to `couch_replicator` to force it to treat all replications as one-shot, rather than continuous (thanks @rnewson): https://www.irccloud.com/pastebin/dFGMLgqm/
* Try the perturbation exercise above after the tweak.

What I'm seeing locally for a TIME_WAIT socket count is, with either approach:
## Unpatched (CouchDB master)
* Starts at ~0, steady state.
* After perturbation, slowly grows to 50-60 (consistent with a 2MSL of 50-60s)
* Never drops back down to 0, even after waiting 5 minutes.

## Patched (Forced one-shot replications)
* Starts at ~6-12, steady state.
* After perturbation, stays around 6-12, steady state.
* Never drops back down to 0, even after waiting 5 minutes.

Contributor guide

Open the contributing guide

Research direction

Start with the couch_replicator area and reproduce the scheduler churn using dev/run, the six replication documents, and the replicator configuration commands described here. Done means rotating a continuous job closes the client-side socket correctly and the reproduced TIME_WAIT count no longer grows as jobs are rotated.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
databases, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.