etcd-io / etcd-io/etcd

High memory usage (memory leak?) with 1 node cluster and small DB (but a lot of updates)

Open
#18,382 18 comments 5 reactions 0 assignees View on GitHub
area/performance priority/awaiting-more-evidence stage/tracked type/bug
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

### Bug report criteria

- [X] This bug report is not security related, security issues should be disclosed privately via [etcd maintainers](mailto:etcd-maintainers@googlegroups.com).
- [X] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [X] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [X] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

My etcd 3.5.8 ate 12 GB of RAM...
The database size is < 2 MB (`etcdctl get --write-out=json --prefix /` gives me 1966139 byte dump).
The database takes 1.5 GB on disk (`/var/lib/etcd0.etcd`).
There are 6 active streams.
Client traffic shows 2.2 MB/s in and 2.5 MB/s out in monitoring dashboard.
There are 128+5 keys which are bumped continuosly through leases.
There are other 139 keys which are updated every 30 seconds.

![etcd_obozhralsya](https://github.com/user-attachments/assets/fc7d4be5-d0a2-4385-a5af-10f141f8765a)

As I've already experienced such memory usage with this etcd, it was started with --enable-pprof=true, pprof shows the following:
```
# go tool pprof http://192.168.1.144:2379/debug/pprof/heap
Fetching profile over HTTP from http://192.168.1.144:2379/debug/pprof/heap
Saved profile in /root/pprof/pprof.etcd.alloc_objects.alloc_space.inuse_objects.inuse_space.012.pb.gz
File: etcd
Type: inuse_space
Time: Jul 31, 2024 at 1:45am (MSK)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 10.88GB, 99.59% of 10.92GB total
Dropped 179 nodes (cum <= 0.05GB)
Showing top 10 nodes out of 17
flat flat% sum% cum cum%
10.88GB 99.59% 99.59% 10.88GB 99.59% go.etcd.io/etcd/api/v3/etcdserverpb.(*InternalRaftRequest).Marshal
0 0% 99.59% 10.88GB 99.59% github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
0 0% 99.59% 10.88GB 99.59% github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
0 0% 99.59% 10.88GB 99.59% github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/api/v3/etcdserverpb._KV_Txn_Handler
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/api/v3/etcdserverpb._KV_Txn_Handler.func1
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).Txn
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).processInternalRaftRequestOnce
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).raftRequest (inline)
0 0% 99.59% 10.88GB 99.59% go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).raftRequestOnce
```

Is it a memory leak or what?...

### What did you expect to happen?

I expected etcd to consume at most 1 GB of memory with 2 MB database

### How can we reproduce it (as minimally and precisely as possible)?

Install Vitastor on 1 node. :-) but I'm not sure it will be reproduced easily.

### Anything else we need to know?

It's still running, I can collect other information or connect with debugger if you tell me what I should collect. :-)

### Etcd version (please run commands below)

```console
$ etcd --version
etcd Version: 3.5.8
Git SHA: 217d183e5
Go Version: go1.19.5
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.5.8
API version: 3.5
```

### Etcd configuration (command line flags or environment variables)

Startup options:

`etcd -name etcd0 --data-dir /var/lib/etcd0.etcd --enable-pprof=true --snapshot-count=1000 --advertise-client-urls http://192.168.1.144:2379 --listen-client-urls http://192.168.1.144:2379 --initial-advertise-peer-urls http://192.168.1.144:2380 --listen-peer-urls http://192.168.1.144:2380 --initial-cluster-token vitastor-etcd-1 --initial-cluster etcd0=http://192.168.1.144:2380 --initial-cluster-state new --max-txn-ops=100000 --max-request-bytes=104857600 --auto-compaction-retention=10 --auto-compaction-mode=revision`

### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

```console
$ etcdctl member list -w table
+------------------+---------+-------+---------------------------+---------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+-------+---------------------------+---------------------------+------------+
| 720141fbb7123c6f | started | etcd0 | http://192.168.1.144:2380 | http://192.168.1.144:2379 | false |
+------------------+---------+-------+---------------------------+---------------------------+------------+

$ etcdctl --endpoints= endpoint status -w table
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.144:2379 | 720141fbb7123c6f | 3.5.8 | 1.2 GB | true | false | 51 | 16960478 | 16960478 | |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
```

### Relevant log output

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the etcd 3.5.8 configuration and the pprof heap profile, focusing on etcdserver.(*EtcdServer).Txn, processInternalRaftRequestOnce, raftRequestOnce, and InternalRaftRequest.Marshal. Reproduce the workload if possible and compare heap behavior while collecting the additional debugger or profile information requested in the issue. Done means the memory-growth cause is identified and the issue has a verified fix or clear resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.