influxdata / influxdata/influxdb
poor use of resources
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Not sure how to reproduce the issue, but seems to be a common problem. Had a raspberry pi running for about 6 months, and just failed. Influxdb wasn't able to launch properly, since it gobbled all the ram available. Every time, it started working its way through the wal and tsm files, until it couldn't request more memory. Had to open the db in a computer, for influx to be able to process all the files.
However, throughout this, influxdb worked great, since I believe no data was lost. Recovered the data and created a backup. Restored the data, rebuilt the indices (buildtsi), and dumptsm. This was on the computer, then moved the sd card back to the rpi.
The /var/lib/influx directory went from 1.5GB to about 300 MB. Also the ram requirements went from a bit under 3GB on the computer (on the rpi it topped at around 2G), to about 1.2GB of memory.
__Expected behavior:__
Either have a built in mechanism to allows it to open or a way to set it into an emergency mode to be able to process the files in batches. Also doesn't seem to process old files well, since 1.2GB went away after trying to recover the db. Seems that it is keeping too many files that should had been rolled into the tsi files.
Would guess many people are using low capacity devices (like rpi), so trying to conserve resources might be vital for many users.
__Actual behavior:__
Aborted the launch with an out of memory message
__Environment info:__
Raspberry Pi 4B with 4GB (Linux 4.19.118-v7l+ armv7l)
Started with 1.6.2(?) and upgraded to 1.8.
__Config:__
was the default setup
Though I did enabled these, in hopes it will help avert this problem
[meta]
dir = "/var/lib/influxdb/meta"
[data]
dir = "/var/lib/influxdb/data"
wal-dir = "/var/lib/influxdb/wal"
index-version = "tsi1"
query-log-enabled = false
cache-snapshot-memory-size = "64m"
cache-snapshot-write-cold-duration = "15m"
compact-full-write-cold-duration = "4h"
max-concurrent-compactions = 1
compact-throughput = "24m"
compact-throughput-burst = "48m"
max-index-log-file-size = "1m"
Contributor guide
Research direction
Start by reproducing recovery on a Raspberry Pi 4B with the reported InfluxDB configuration, focusing on WAL and TSM processing and memory use. Compare recovery on the Pi with the computer-based recovery described in the report. Done would mean recovery completes within the device's available memory or provides a documented emergency or batch-processing mode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, raspberry-pi
- Domain
- databases, embedded-iot, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100