apache / apache/couchdb

Increasing memory usage with debug log level and noisy logs from reduce functions

Open
#5,474 1 comment 1 reaction 0 assignees View on GitHub
bug needs-triage
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

## Description

When log level is set to `debug`, there exists at least one view with "lengthy" emissions (many characters, for example uuids) with a reduce function, and there is a constant inflow of documents, CouchDb will gradually increase RAM usage, until it runs out of memory.

## Steps to Reproduce

This can be replicated on latest version of CouchDb (I just launched a local instance in a docker container) in single node.

In this gist https://gist.github.com/dianabarsan/408ed8b08bb653a6d16c04ba2126556b, there is a script that:
- sets log level to debug
- creates a database with one ddoc with one reduce function
- pushes 1 million documents that would be indexed
- triggers view indexes

Running this script against a CouchDb in a docker container (compose.yml also provided in the gist) should trigger gradually increasing RAM usage.

## Expected Behaviour

Memory should be released, even when log level is debug.

## Your Environment

Production server: docker image of CouchDb v. 3.3.3 launched in Kubernetes cluster hosted on AWS

Local test server: docker image of CouchDb 3.4.2, launched in docker container on Manjaro Linux.

## Additional Context

In a three node CouchDb cluster, we have observed one node gradually increasing RAM usage compared to the other two nodes - up to 200GB vs 3GB. The k8s node would eventually run out of RAM and be expelled.
Our deployment receives a rather constant flow of new documents.

The only difference between the cluster nodes was that the runaway RAM node had the log level set to `debug`.

Upon further investigation, it turned out that having reduce functions (in our case _stats) for views that have "lengthy" emissions was triggering the high RAM usage, presumably due to accumulating large debug logs.

After reverting the log level to `info`, this behavior disappeared and RAM usage became consistent within the CouchDb cluster.

For my local deployment, I have noticed that RAM does _eventually_ get released gradually AFTER there are no more writes. In our production environment this was never the case.

Contributor guide

Open the contributing guide

Research direction

Start by running the linked gist and its compose.yml against the reported CouchDB versions, comparing debug and info logging while the reduce view processes continuous writes. Trace the reduce-function logging path and verify that debug-level reproduction no longer shows steadily increasing memory usage under sustained indexing.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.