influxdata / influxdata/influxdb

Reboot after OOM error enters setup mode, wiping all users & tokens

Open
#24,891 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behaviour.

1. Create a query which uses up a lot of memory (`|> group() |> count() |> group() |> sort()` or alike)
2. Set a timeframe for the query to consume a large amount of data, in my case:
- 150 rows of data
- every 10 seconds
- 1 hour timeframe
- transform 150 rows to a single row, then count distinct filtered rows
3. Wait for server to consume too much memory, crashing the server

__Expected behaviour:__
After the crash (`Apr 3 08:21:42 - influxd-systemd-start.sh: fatal error: runtime: out of memory`) I expect the server to recovery to the same data and have a tiny gap of data due to the influxd process getting killed and respawned.

__Actual behaviour:__
Influxd rebooted and went into setup mode, seemingly recreating the `influxd.bolt` (`stat.btime = Wed Apr 3 08:21:49 2024`) and forgetting the existance of all the users, tokens, etc.

__Environment info:__

* Install, ansible information:
```
# defaults file for influxdb
influxdb_version: 2.7.5-1
influxdb_client_version: 2.7.3
```
and the playbook:
```
---
# tasks file for influxdb
- name: Upgrade all packages # noqa: package-latest
ansible.builtin.yum:
name: '*'
state: latest

- name: Create directory
ansible.builtin.file:
path: /tmp/influx_client
state: directory

- name: Download InfluxDB Client
ansible.builtin.get_url:
url: "https://dl.influxdata.com/influxdb/releases/influxdb2-client-{{ influxdb_client_version }}-linux-amd64.tar.gz"
dest: /tmp/influx_client
owner: root
group: wheel
mode: 0755

- name: Extract Influxdb client
ansible.builtin.unarchive:
src: "/tmp/influx_client/influxdb2-client-{{ influxdb_client_version }}-linux-amd64.tar.gz"
dest: /usr/bin/
mode: '0755'

- name: Download InfluxDB
ansible.builtin.get_url:
url: "https://dl.influxdata.com/influxdb/releases/influxdb2-{{ influxdb_version }}.x86_64.rpm"
dest: /tmp/
owner: root
group: wheel
mode: 0644

- name: Install Influxdb
ansible.builtin.yum:
name: "/tmp/influxdb2-{{ influxdb_version }}.x86_64.rpm"
state: present

- name: Start service influxdb
ansible.builtin.service:
name: influxdb
state: started
enabled: yes

- name: Run setup
ansible.builtin.command: influx setup --username {{ username }} --password {{ password }} --token {{ token }} --org {{ organization }} --bucket {{ bucket1 }} --force

- name: Run command
ansible.builtin.command: influx bucket create --skip-verify -name {{ bucket2 }} --org {{ organization }} --host=http://localhost:8086 --token {{ token }}

- name: Stop service influxdb
ansible.builtin.service:
name: influxdb
state: stopped
```
* System info: `Linux 4.14.336-257.566.amzn2.x86_64 x86_64`
* ec2 aws setup, t3.large, no other processes on the server of importance, 1 data collector which pushes the data to influxdb. However this stops working as the tokens are purged on the crash.

__Logs:__
Can be posted upon request, but it's basically an oom error and then multiple page of go stacktrace.

Reboot from journalctl:
```
pr 03 08:21:48 - systemd[1]: influxdb.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 03 08:21:48 - systemd[1]: Unit influxdb.service entered failed state.
Apr 03 08:21:48 - systemd[1]: influxdb.service failed.
Apr 03 08:21:48 - systemd[1]: influxdb.service holdoff time over, scheduling restart.
Apr 03 08:21:48 - systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Apr 03 08:21:48 - systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: Command "print-config" is deprecated, use the influx-cli command server-config
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: Command "print-config" is deprecated, use the influx-cli command server-config
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.318930Z lvl=info msg="Welcome to InfluxDB" log_id=0oKldn
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.338316Z lvl=info msg="Resources opened" log_id=0oKldnWW0
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.338470Z lvl=info msg="Resources opened" log_id=0oKldnWW0
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.341452Z lvl=info msg="Bringing up metadata migrations" l
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: Command "print-config" is deprecated, use the influx-cli command server-config
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.512177Z lvl=info msg="Bringing up metadata migrations" l
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.574469Z lvl=info msg="Using data dir" log_id=0oKldnWW000
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.574507Z lvl=info msg="Compaction settings" log_id=0oKldn
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.574518Z lvl=info msg="Open store (start)" log_id=0oKldnW
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.619038Z lvl=info msg="index opened with 8 partitions" lo
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.621538Z lvl=info msg="loading changes (start)" log_id=0o
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.624846Z lvl=info msg="index opened with 8 partitions" lo
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.629590Z lvl=info msg="loading changes (start)" log_id=0o
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.636437Z lvl=info msg="loading changes (end)" log_id=0oKl
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.651998Z lvl=info msg="loading changes (end)" log_id=0oKl
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.647011Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.660826Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.676034Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.687784Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.688511Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.694594Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.694763Z lvl=info msg="Opened shard" log_id=0oKldnWW000 s
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.696700Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.696828Z lvl=info msg="Reading file" log_id=0oKldnWW000 s
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.728168Z lvl=info msg="index opened with 8 partitions" lo
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.729435Z lvl=info msg="loading changes (start)" log_id=0o
lines 49-96
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the out-of-memory query and reviewing the systemd restart logs around influxd.bolt creation and setup mode. Trace why a restarted server does not recover its existing users and tokens; done means an OOM-triggered restart preserves the prior metadata and requires no new setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, linux
Domain
database
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.