High memory usage (memory leak?) with 1 node cluster and small DB (but a lot of updates)
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
### Bug report criteria
- [X] This bug report is not security related, security issues should be disclosed privately via [etcd maintainers](mailto:etcd-maintainers@googlegroups.com).
- [X] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [X] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [X] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.
### What happened?
My etcd 3.5.8 ate 12 GB of RAM...
The database size is < 2 MB (`etcdctl get --write-out=json --prefix /` gives me 1966139 byte dump).
The database takes 1.5 GB on disk (`/var/lib/etcd0.etcd`).
There are 6 active streams.
Client traffic shows 2.2 MB/s in and 2.5 MB/s out in monitoring dashboard.
There are 128+5 keys which are bumped continuosly through leases.
There are other 139 keys which are updated every 30 seconds.

As I've already experienced such memory usage with this etcd, it was started with --enable-pprof=true, pprof shows the following:
```
# go tool pprof http://192.168.1.144:2379/debug/pprof/heap
Fetching profile over HTTP from http://192.168.1.144:2379/debug/pprof/heap
Saved profile in /root/pprof/pprof.etcd.alloc_objects.alloc_space.inuse_objects.inuse_space.012.pb.gz
File: etcd
Type: inuse_space
Time: Jul 31, 2024 at 1:45am (MSK)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 10.88GB, 99.59% of 10.92GB total
Dropped 179 nodes (cum <= 0.05GB)
Showing top 10 nodes out of 17
flat flat% sum% cum cum%
10.88GB 99.59% 99.59% 10.88GB 99.59% go.etcd.io/etcd/api/v3/etcdserverpb.(*InternalRaftRequest).Marshal
0 0% 99.59% 10.88GB 99.59% github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
0 0% 99.59% 10.88GB 99.59% github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
0 0% 99.59% 10.88GB 99.59% github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/api/v3/etcdserverpb._KV_Txn_Handler
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/api/v3/etcdserverpb._KV_Txn_Handler.func1
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).Txn
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).processInternalRaftRequestOnce
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).raftRequest (inline)
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).raftRequestOnce
```
Is it a memory leak or what?...
### What did you expect to happen?
I expected etcd to consume at most 1 GB of memory with 2 MB database
### How can we reproduce it (as minimally and precisely as possible)?
Install Vitastor on 1 node. :-) but I'm not sure it will be reproduced easily.
### Anything else we need to know?
It's still running, I can collect other information or connect with debugger if you tell me what I should collect. :-)
### Etcd version (please run commands below)
```console
$ etcd --version
etcd Version: 3.5.8
Git SHA: 217d183e5
Go Version: go1.19.5
Go OS/Arch: linux/amd64
$ etcdctl version
etcdctl version: 3.5.8
API version: 3.5
```
### Etcd configuration (command line flags or environment variables)
Startup options:
`etcd -name etcd0 --data-dir /var/lib/etcd0.etcd --enable-pprof=true --snapshot-count=1000 --advertise-client-urls http://192.168.1.144:2379 --listen-client-urls http://192.168.1.144:2379 --initial-advertise-peer-urls http://192.168.1.144:2380 --listen-peer-urls http://192.168.1.144:2380 --initial-cluster-token vitastor-etcd-1 --initial-cluster etcd0=http://192.168.1.144:2380 --initial-cluster-state new --max-txn-ops=100000 --max-request-bytes=104857600 --auto-compaction-retention=10 --auto-compaction-mode=revision`
### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
```console
$ etcdctl member list -w table
+------------------+---------+-------+---------------------------+---------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+-------+---------------------------+---------------------------+------------+
| 720141fbb7123c6f | started | etcd0 | http://192.168.1.144:2380 | http://192.168.1.144:2379 | false |
+------------------+---------+-------+---------------------------+---------------------------+------------+
$ etcdctl --endpoints= endpoint status -w table
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.144:2379 | 720141fbb7123c6f | 3.5.8 | 1.2 GB | true | false | 51 | 16960478 | 16960478 | |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
```
### Relevant log output
_No response_
Contributor guide
Research direction
Start with the etcd 3.5.8 configuration and the pprof heap profile, focusing on etcdserver.(*EtcdServer).Txn, processInternalRaftRequestOnce, raftRequestOnce, and InternalRaftRequest.Marshal. Reproduce the workload if possible and compare heap behavior while collecting the additional debugger or profile information requested in the issue. Done means the memory-growth cause is identified and the issue has a verified fix or clear resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100