Increasing memory usage with debug log level and noisy logs from reduce functions
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
## Description
When log level is set to `debug`, there exists at least one view with "lengthy" emissions (many characters, for example uuids) with a reduce function, and there is a constant inflow of documents, CouchDb will gradually increase RAM usage, until it runs out of memory.
## Steps to Reproduce
This can be replicated on latest version of CouchDb (I just launched a local instance in a docker container) in single node.
In this gist https://gist.github.com/dianabarsan/408ed8b08bb653a6d16c04ba2126556b, there is a script that:
- sets log level to debug
- creates a database with one ddoc with one reduce function
- pushes 1 million documents that would be indexed
- triggers view indexes
Running this script against a CouchDb in a docker container (compose.yml also provided in the gist) should trigger gradually increasing RAM usage.
## Expected Behaviour
Memory should be released, even when log level is debug.
## Your Environment
Production server: docker image of CouchDb v. 3.3.3 launched in Kubernetes cluster hosted on AWS
Local test server: docker image of CouchDb 3.4.2, launched in docker container on Manjaro Linux.
## Additional Context
In a three node CouchDb cluster, we have observed one node gradually increasing RAM usage compared to the other two nodes - up to 200GB vs 3GB. The k8s node would eventually run out of RAM and be expelled.
Our deployment receives a rather constant flow of new documents.
The only difference between the cluster nodes was that the runaway RAM node had the log level set to `debug`.
Upon further investigation, it turned out that having reduce functions (in our case _stats) for views that have "lengthy" emissions was triggering the high RAM usage, presumably due to accumulating large debug logs.
After reverting the log level to `info`, this behavior disappeared and RAM usage became consistent within the CouchDb cluster.
For my local deployment, I have noticed that RAM does _eventually_ get released gradually AFTER there are no more writes. In our production environment this was never the case.
Contributor guide
Research direction
Start by running the linked gist and its compose.yml against the reported CouchDB versions, comparing debug and info logging while the reduce view processes continuous writes. Trace the reduce-function logging path and verify that debug-level reproduction no longer shows steadily increasing memory usage under sustained indexing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100