Database replication with filter function might timeout in case of lack of filter hits
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
## Description
I am bringing this discussion from the CouchDB slack channel and it's still to be decided whether it's really a bug.
So, I started a database replication of a database that has ~3M of deleted documents and only ~60k non_deleted ones (the database reports around 37M update_seq). The replication document is defined with a javascript filter function which will skip deleted documents.
Checking the logs of couchdb, I noticed that the replication was failing and starting over and over. Error reported in the log is something like:
```
Replicator, request GET to "source_url_db/_changes?filter=ddoc_with_filter&feed=continuous&style=all_docs&since=0&timeout=100000" failed due to error req_timedout
```
My understand is that it's actually timing out because there was no documents passing the filter during that 100secs timeout, thus failing the _changes request.
## Steps to Reproduce
Execute a replication task for a source database where the vast majority of documents are deleted.
1. Create a design document with filter function in the source node:
```
curl -ks -X POST http://URL:5984/source_db/_design/WorkQueue -H 'Content-Type:application/json' -d '{"filters": {"filterDeletedDocs": function(doc, req) {return !doc._deleted;}}}'
```
2. in the target node (running replicator), we create a replication document:
```
curl -ks -X POST http://URL:5984/_replicator -H 'Content-Type:application/json' -d '{"_id":"rep_wq", "source":"https://URL/couchdb/source_db/", "target":"http://URL:5984/target_db/", "continuous": true, "filter":"WorkQueue/filterDeletedDocs"}'
```
## Expected Behaviour
I would expect the replication not to timeout in case it takes longer than the timeout defined in the replication (or in the replicator configuration) to find documents matching the replication filter. In other words, if the source CouchDB instance is doing heavy readings to replicate data, it should be considered as normal and no timeout should be raised.
## Your Environment
Source database is running CouchDB 1.6.1 (hopefully soon to be upgraded!)
Target database (running replicator) is running CouchDB 3.1.2:
```
{"couchdb":"Welcome","version":"3.1.2","git_sha":"572b68e72","uuid":"7890c19e733322170739ab2520eb4ab1","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
```
The replicator/target configuration is configured with:
```
[replicator]
use_checkpoints = true
checkpoint_interval = 120000
worker_processes = 4
http_connections = 10
worker_batch_size = 2000
socket_options = [{keepalive, true}, {nodelay, true}]
max_replication_retry_count = infinity
connection_timeout = 900000
```
I also tested replication with the following changes (with no success):
```
[chttpd]
changes_timeout = 300000
```
Lastly, here is a snapshot of the database I was trying to replicate from:
```
{"db_name":"workqueue","doc_count":61306,"doc_del_count":3345978,"update_seq":37812510,"purge_seq":0,"compact_running":false,"disk_size":8065118340,"data_size":2165447890,"instance_start_time":"1655923205137624","disk_format_version":6,"committed_update_seq":37812510}
```
Regarding the OS, it's CentOS Linux 7 (x86_64)
## Additional Context
The _changes timeout is followed with this error:
```
[error] 2022-06-16T22:27:59.320726Z couchdb@127.0.0.1 <0.30721.1> -------- ChangesReader process died with reason: {changes_reader_died,{timeout,ibrowse_stream_cleanup}}
```
which then restarts the replication job.
Contributor guide
Research direction
Start by reproducing the continuous replication using the filtered _changes request and the supplied database statistics, then inspect the replicator's ChangesReader timeout path. Check how the _changes timeout, ChangesReader process, and replicator connection_timeout interact. Done means filtered replication remains healthy when no matching documents are found for longer than the current timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang, javascript
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100