Compaction Failures Similar to #2941 on Upgrade From 2.x to 3.1.1
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
## Description
we're seeing a similar issue to https://github.com/apache/couchdb/issues/2941 with random compactions on shards when upgrading from 2.x to 3.1.1. But might be completely unrelated. The compaction metadata file blew up to about 500gb over 24 hours for a shard that is about 30gb constantly hitting this error. Similarly we're also seeing large disk io on the nodes this is happening on versus nodes this is not happening on like in #2941. I've deleted the compaction files like discussed in the previous issue and seems to be working fine now, the compaction is running and the errors have stopped. disk io has gone back down.
`<0.5181.0> -------- exit for compaction of ["shards/60000000-7fffffff/core_activations.1589336264"]: {badarith,[{couch_file,get_pread_locnum,3,[{file,"src/couch_file.erl"},{line,730}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{couch_file,read_multi_raw_iolists_int,2,[{file,"src/couch_file.erl"},{line,719}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,507}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,636}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,665}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}`
`-------- CRASH REPORT Process (<0.32443.1691>) with 3 neighbors crashed with reason: bad arithmetic expression at couch_file:get_pread_locnum/3(line:730) <= lists:map/2(line:1239) <= couch_file:read_multi_raw_iolists_int/2(line:719) <= couch_file:handle_call/3(line:507) <= gen_server:try_handle_call/4(line:636) <= gen_server:handle_msg/6(line:665) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.3352.1692>], message_queue_len: 0, messages: [], links: [<0.3352.1692>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.1924847>,92}},"..."}},...], trap_exit: false, status: running, heap_size: 28690, stack_size: 27, reductions: 13483`
an example of a shard I haven't cleaned up yet
`-rw-r--r-- 1 1501 1501 16G Dec 9 17:13 core_activations.1589323247.couch`
`-rw-r--r-- 1 1501 1501 4.0G Dec 9 17:11 core_activations.1589323247.couch.compact.data`
`-rw-r--r-- 1 1501 1501 159G Dec 9 17:13 core_activations.1589323247.couch.compact.meta`
## Steps to Reproduce
Upgrade to 3.1.1 from 2.x mid compaction
## Expected Behaviour
Compaction to complete as expected
## Your Environment
If any specific environment details would be helpful just let me know
* CouchDB version used: 3.1.1
* Browser name and version:
* Operating system and version: centOS
Contributor guide
Research direction
Start with src/couch_file.erl at get_pread_locnum/3 (line 730), read_multi_raw_iolists_int/2 (line 719), and handle_call/3 (line 507), then trace the compaction path shown in the crash report. Reproduce an upgrade from CouchDB 2.x to 3.1.1 during compaction and inspect the .compact.meta and .compact.data files. Done means compaction completes without badarith crashes or uncontrolled metadata growth.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100