apache / apache/couchdb

couchspawnkillable: Cannot allocate memory

Open
#4,988 2 comments 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

## Description

I use `apache/couchdb:3.3.2` docker image on my server.
Today couchdb process crashed because no free memory on the server.
CouchDB logs contians

> -- Logs begin at Tue 2024-01-09 16:32:03 UTC, end at Tue 2024-02-20 22:09:12 UTC. --
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory/opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memorysh: 1: exec: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: ^M
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec:
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:50:07 ippbx-0a.example.com docker[999209]: out of memory
Feb 20 21:06:23 ippbx-0a.example.com docker[999209]: out of memory
Feb 20 21:10:08 ippbx-0a.example.com docker[999209]: eheap_alloc: Cannot allocate 71302344 bytes of memory (of type "heap").
Feb 20 21:10:08 ippbx-0a.example.com docker[999209]: ^M^M
Feb 20 21:19:07 ippbx-0a.example.com docker[999209]: Crash dump is being written to: erl_crash.dump...
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Main process exited, code=exited, status=137/n/a
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Failed with result 'exit-code'.
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Service RestartSec=100ms expired, scheduling restart.
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Scheduled restart job, restart counter is at 2.
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: Stopped CouchDB container.

## Steps to Reproduce

Do not know

## Expected Behaviour

All external processes should not lead to all server memory usage.

## Your Environment

* CouchDB version used: 3.3.2
* Browser name and version: do not used
* Operating system and version: CentOS 8 Stream, aarch64

## Additional Context

Here is graph for swap memory usage on the server. Swap usage start grow about one week ago
![image](https://github.com/apache/couchdb/assets/2562241/d87aa61b-4602-4688-9841-265c00c0204e)

And the disk cache started shrinking about 8 days ago
![image](https://github.com/apache/couchdb/assets/2562241/e3103dd1-5438-478b-9487-d9015b9f1971)

A related ticket found at https://issues.apache.org/jira/browse/COUCHDB-371 and looks like issue is not solved and closed with comment
> If this issue remains in CouchDB 2.x, please file a new issue

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with the apache/couchdb:3.3.2 Docker image on CentOS 8 Stream aarch64 and inspect the reported couchspawnkillable, out-of-memory, and Erlang crash-dump logs. No source file or test is named; done would require a reproducible case and evidence that external processes cannot exhaust all server memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, erlang
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.