netdata state container: runaway FD use
Open
Nobody has claimed this yet.
- Dominant language
- Go Template
- Stars
- 177
- Forks
- 81
- Avg merge
- 5h 48m
- Merged PRs (30d)
- 5
Description
Netdata needed to be removed due to consuming ALL available file descriptors (mildly entertaining that this isn't a monitored metric in netdata, that I could find).
From lsof it appears that it is just opening the WAL/db in a loop:
netdata 1781992 1782297 RRDCONTEX 201 15r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 16u sock 0,8 0t0 62179404 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 17ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 18u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 19ur REG 0,121 32768 14044312 /var/cache/netdata/netdata-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 20ur REG 0,121 4096 14044313 /var/cache/netdata/context-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 21u REG 0,121 37112 14044314 /var/cache/netdata/context-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 22ur REG 0,121 32768 14044315 /var/cache/netdata/context-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 23ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 24u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 25u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 26r FIFO 0,13 0t0 62177496 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 27w FIFO 0,13 0t0 62177496 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 28u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 29u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 30r FIFO 0,13 0t0 62171736 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 31w FIFO 0,13 0t0 62171736 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 32u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 33r FIFO 0,13 0t0 62176463 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 34u sock 0,8 0t0 62161843 protocol: UDPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 35w FIFO 0,13 0t0 62166944 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 36u sock 0,8 0t0 62161844 protocol: UDP
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 37u sock 0,8 0t0 62161847 protocol: TCPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 38u sock 0,8 0t0 62161848 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 39u sock 0,8 0t0 62178506 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 40u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 41u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 42r FIFO 0,13 0t0 62178505 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 43w FIFO 0,13 0t0 62178505 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 44r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 45r FIFO 0,13 0t0 62160878 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 46w FIFO 0,13 0t0 62160878 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 47r REG 0,273 0 62169795 /proc/1/task/209/stat
lsof: no pwd entry for UID 201
netdata 1781992 1782297 RRDCONTEX 201 48u sock 0,8 0t0 62194554 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 cwd DIR 0,121 4096 13901928 /etc/netdata
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 rtd DIR 0,121 4096 14034715 /
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 txt REG 0,121 6986240 13902420 /usr/sbin/netdata
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem-r REG 0,121 14044315 /var/cache/netdata/context-meta.db-shm (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem-r REG 0,121 14044312 /var/cache/netdata/netdata-meta.db-shm (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896738 /usr/lib/libzstd.so.1.5.5 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896436 /usr/lib/libgcc_s.so.1 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896627 /usr/lib/libstdc++.so.6.0.30 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896405 /usr/lib/libbson-1.0.so.0.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896522 /usr/lib/libmongoc-1.0.so.0.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896615 /usr/lib/libsnappy.so.1.1.10 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896595 /usr/lib/libprotobuf.so.32.0.12 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896491 /usr/lib/libjson-c.so.5.2.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896512 /usr/lib/liblz4.so.1.9.4 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13896695 /usr/lib/libuv.so.1.0.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 13895904 /lib/libuuid.so.1.3.0 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672957 /lib/libssl.so.3 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672956 /lib/libcrypto.so.3 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672959 /lib/libz.so.1.2.13 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 mem REG 0,121 11672951 /lib/ld-musl-x86_64.so.1 (stat: No such file or directory)
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 0w CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 1w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 2w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 3u sock 0,8 0t0 62173481 protocol: UNIX
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 4w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 5w FIFO 0,13 0t0 62178474 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 6w REG 0,121 0 14044307 /var/log/netdata/health.log
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 7u sock 0,8 0t0 62173487 protocol: TCP
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 8u sock 0,8 0t0 62173488 protocol: TCPv6
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 9u a_inode 0,14 0 12713 [eventpoll]
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 10r FIFO 0,13 0t0 62179402 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 11w FIFO 0,13 0t0 62179402 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 12r FIFO 0,13 0t0 62179403 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 13w FIFO 0,13 0t0 62179403 pipe
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 14u a_inode 0,14 0 12713 [eventfd]
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 15r CHR 1,3 0t0 6 /dev/null
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 16u sock 0,8 0t0 62179404 protocol: UNIX-STREAM
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 17ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 18u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 19ur REG 0,121 32768 14044312 /var/cache/netdata/netdata-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 20ur REG 0,121 4096 14044313 /var/cache/netdata/context-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 21u REG 0,121 37112 14044314 /var/cache/netdata/context-meta.db-wal
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 22ur REG 0,121 32768 14044315 /var/cache/netdata/context-meta.db-shm
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 23ur REG 0,121 4587520 14044310 /var/cache/netdata/netdata-meta.db
lsof: no pwd entry for UID 201
netdata 1781992 1782298 REPLAY[1] 201 24u REG 0,121 4906952 14044311 /var/cache/netdata/netdata-meta.db-wal
approximately ^ 100's of thousands of times until the node eventually becomes unavailable due to an inability to open any more file descriptors.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Netdata state-container deployment and inspect how the Helm chart configures its WAL and database paths. Use the lsof output in the issue to verify whether repeated netdata-meta.db and context-meta.db handles are opened; done means the state container no longer exhausts available file descriptors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100