apache / apache/couchdb

Replication crashes just on single database from many

Open
#4,204 8 comments 1 reaction 0 assignees View on GitHub
bug needs-triage
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

Couchdb 2.3.1 on Centos7

I'm replicating more than 20dbs from one server to another. The process works flawlessly apart from a single database.
The database in question consists of more than 100k small documents like below:

`
"db_name": "test",
"purge_seq": "0-g1AAAAFTeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMeC5BkeACk_gNBViIDHrVJCUAyqZ6gOoiZCyBm7idG7QGI2vsE7FcA2W9P0P5EhiR5wp5xABkWT6RnGiAOnA9UmwUAtixejg",
"update_seq": "100785-g1AAAAFreJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____s5IYGAyr8KhLUgCSSfYwpdX4lDqAlMZDlRrswqc0AaS0HmaqPx6leSxAkqEBSAFVzwcr_0FQ-QKI8v1gh_wlqPwARPl9sOnsBJU_gCiHeHN7FgAkvmTM",
"sizes": {
"file": 82658990,
"external": 77019483,
"active": 82112519
},
"other": {
"data_size": 77019483
},
"doc_del_count": 4,
"doc_count": 100766,
"disk_size": 82658990,
"disk_format_version": 7,
"data_size": 82112519,
"compact_running": false,
"cluster": {
"q": 8,
"n": 1,
"w": 1,
"r": 1
},
"instance_start_time": "0"
`

After replicating about 79k docs, the replication crashes with output like below:

`[error] 2022-10-12T10:43:26.563124Z couchdb@127.0.0.1 <0.30702.7> -------- CRASH REPORT Process (<0.30702.7>) with 5 neighbors exited with reason: {worker_died,<0.30700.7>,{process_died,<0.3280.8>,{{nocatch,missing_doc},[{couch_replicator_api_wrap,open_doc_revs,6,[{file,"src/couch_replicator_api_wrap.erl"},{line,302}]},{couch_replicator_worker,'-spawn_doc_reader/3-fun-1-',4,[{file,"src/couch_replicator_worker.erl"},{line,323}]}]}}} at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.30607.7>,couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [], links: [<0.3401.8>,<0.3503.8>,<0.3700.8>,<0.3413.8>,<0.30703.7>], dictionary: [{last_stats_report,{1665,571404,580133}}], trap_exit: true, status: running, heap_size: 6772, stack_size: 27, reductions: 77732`

When I copied this db manually to the second server, there is no problem anymore. Can add documents on main server and they are being copied to the second one.

What is a possible culprit of this issue?

Contributor guide

Open the contributing guide

Research direction

Start with src/couch_replicator_api_wrap.erl at line 302 and src/couch_replicator_worker.erl at line 323, then reproduce replication for the large database described in the report. Trace the missing_doc crash and determine why this document batch terminates the worker. Done means replication no longer crashes for this case and the failure has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.