influxdata / influxdata/influxdb
Reboot after OOM error enters setup mode, wiping all users & tokens
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behaviour.
1. Create a query which uses up a lot of memory (`|> group() |> count() |> group() |> sort()` or alike)
2. Set a timeframe for the query to consume a large amount of data, in my case:
- 150 rows of data
- every 10 seconds
- 1 hour timeframe
- transform 150 rows to a single row, then count distinct filtered rows
3. Wait for server to consume too much memory, crashing the server
__Expected behaviour:__
After the crash (`Apr 3 08:21:42 - influxd-systemd-start.sh: fatal error: runtime: out of memory`) I expect the server to recovery to the same data and have a tiny gap of data due to the influxd process getting killed and respawned.
__Actual behaviour:__
Influxd rebooted and went into setup mode, seemingly recreating the `influxd.bolt` (`stat.btime = Wed Apr 3 08:21:49 2024`) and forgetting the existance of all the users, tokens, etc.
__Environment info:__
* Install, ansible information:
```
# defaults file for influxdb
influxdb_version: 2.7.5-1
influxdb_client_version: 2.7.3
```
and the playbook:
```
---
# tasks file for influxdb
- name: Upgrade all packages # noqa: package-latest
ansible.builtin.yum:
name: '*'
state: latest
- name: Create directory
ansible.builtin.file:
path: /tmp/influx_client
state: directory
- name: Download InfluxDB Client
ansible.builtin.get_url:
url: "https://dl.influxdata.com/influxdb/releases/influxdb2-client-{{ influxdb_client_version }}-linux-amd64.tar.gz"
dest: /tmp/influx_client
owner: root
group: wheel
mode: 0755
- name: Extract Influxdb client
ansible.builtin.unarchive:
src: "/tmp/influx_client/influxdb2-client-{{ influxdb_client_version }}-linux-amd64.tar.gz"
dest: /usr/bin/
mode: '0755'
- name: Download InfluxDB
ansible.builtin.get_url:
url: "https://dl.influxdata.com/influxdb/releases/influxdb2-{{ influxdb_version }}.x86_64.rpm"
dest: /tmp/
owner: root
group: wheel
mode: 0644
- name: Install Influxdb
ansible.builtin.yum:
name: "/tmp/influxdb2-{{ influxdb_version }}.x86_64.rpm"
state: present
- name: Start service influxdb
ansible.builtin.service:
name: influxdb
state: started
enabled: yes
- name: Run setup
ansible.builtin.command: influx setup --username {{ username }} --password {{ password }} --token {{ token }} --org {{ organization }} --bucket {{ bucket1 }} --force
- name: Run command
ansible.builtin.command: influx bucket create --skip-verify -name {{ bucket2 }} --org {{ organization }} --host=http://localhost:8086 --token {{ token }}
- name: Stop service influxdb
ansible.builtin.service:
name: influxdb
state: stopped
```
* System info: `Linux 4.14.336-257.566.amzn2.x86_64 x86_64`
* ec2 aws setup, t3.large, no other processes on the server of importance, 1 data collector which pushes the data to influxdb. However this stops working as the tokens are purged on the crash.
__Logs:__
Can be posted upon request, but it's basically an oom error and then multiple page of go stacktrace.
Reboot from journalctl:
```
pr 03 08:21:48 - systemd[1]: influxdb.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 03 08:21:48 - systemd[1]: Unit influxdb.service entered failed state.
Apr 03 08:21:48 - systemd[1]: influxdb.service failed.
Apr 03 08:21:48 - systemd[1]: influxdb.service holdoff time over, scheduling restart.
Apr 03 08:21:48 - systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Apr 03 08:21:48 - systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: Command "print-config" is deprecated, use the influx-cli command server-config
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: Command "print-config" is deprecated, use the influx-cli command server-config
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.318930Z lvl=info msg="Welcome to InfluxDB" log_id=0oKldn
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.338316Z lvl=info msg="Resources opened" log_id=0oKldnWW0
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.338470Z lvl=info msg="Resources opened" log_id=0oKldnWW0
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.341452Z lvl=info msg="Bringing up metadata migrations" l
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: Command "print-config" is deprecated, use the influx-cli command server-config
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.512177Z lvl=info msg="Bringing up metadata migrations" l
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.574469Z lvl=info msg="Using data dir" log_id=0oKldnWW000
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.574507Z lvl=info msg="Compaction settings" log_id=0oKldn
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.574518Z lvl=info msg="Open store (start)" log_id=0oKldnW
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.619038Z lvl=info msg="index opened with 8 partitions" lo
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.621538Z lvl=info msg="loading changes (start)" log_id=0o
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.624846Z lvl=info msg="index opened with 8 partitions" lo
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.629590Z lvl=info msg="loading changes (start)" log_id=0o
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.636437Z lvl=info msg="loading changes (end)" log_id=0oKl
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.651998Z lvl=info msg="loading changes (end)" log_id=0oKl
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.647011Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.660826Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.676034Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.687784Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.688511Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.694594Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.694763Z lvl=info msg="Opened shard" log_id=0oKldnWW000 s
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.696700Z lvl=info msg="Opened file" log_id=0oKldnWW000 se
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.696828Z lvl=info msg="Reading file" log_id=0oKldnWW000 s
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.728168Z lvl=info msg="index opened with 8 partitions" lo
Apr 03 08:21:49 - influxd-systemd-start.sh[14084]: ts=2024-04-03T08:21:49.729435Z lvl=info msg="loading changes (start)" log_id=0o
lines 49-96
```
Contributor guide
Research direction
Start by reproducing the out-of-memory query and reviewing the systemd restart logs around influxd.bolt creation and setup mode. Trace why a restarted server does not recover its existing users and tokens; done means an OOM-triggered restart preserves the prior metadata and requires no new setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ansible, linux
- Domain
- database
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100