influxdata / influxdata/influxdb

poor use of resources

Open
#18,765 2 comments 0 reactions 0 assignees View on GitHub
1.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Not sure how to reproduce the issue, but seems to be a common problem. Had a raspberry pi running for about 6 months, and just failed. Influxdb wasn't able to launch properly, since it gobbled all the ram available. Every time, it started working its way through the wal and tsm files, until it couldn't request more memory. Had to open the db in a computer, for influx to be able to process all the files.

However, throughout this, influxdb worked great, since I believe no data was lost. Recovered the data and created a backup. Restored the data, rebuilt the indices (buildtsi), and dumptsm. This was on the computer, then moved the sd card back to the rpi.

The /var/lib/influx directory went from 1.5GB to about 300 MB. Also the ram requirements went from a bit under 3GB on the computer (on the rpi it topped at around 2G), to about 1.2GB of memory.

__Expected behavior:__
Either have a built in mechanism to allows it to open or a way to set it into an emergency mode to be able to process the files in batches. Also doesn't seem to process old files well, since 1.2GB went away after trying to recover the db. Seems that it is keeping too many files that should had been rolled into the tsi files.

Would guess many people are using low capacity devices (like rpi), so trying to conserve resources might be vital for many users.

__Actual behavior:__
Aborted the launch with an out of memory message

__Environment info:__
Raspberry Pi 4B with 4GB (Linux 4.19.118-v7l+ armv7l)
Started with 1.6.2(?) and upgraded to 1.8.

__Config:__
was the default setup

Though I did enabled these, in hopes it will help avert this problem
[meta]
dir = "/var/lib/influxdb/meta"
[data]
dir = "/var/lib/influxdb/data"
wal-dir = "/var/lib/influxdb/wal"
index-version = "tsi1"
query-log-enabled = false
cache-snapshot-memory-size = "64m"
cache-snapshot-write-cold-duration = "15m"
compact-full-write-cold-duration = "4h"
max-concurrent-compactions = 1
compact-throughput = "24m"
compact-throughput-burst = "48m"
max-index-log-file-size = "1m"

Contributor guide

Open the contributing guide

Research direction

Start by reproducing recovery on a Raspberry Pi 4B with the reported InfluxDB configuration, focusing on WAL and TSM processing and memory use. Compare recovery on the Pi with the computer-based recovery described in the report. Done would mean recovery completes within the device's available memory or provides a documented emergency or batch-processing mode.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, raspberry-pi
Domain
databases, embedded-iot, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.