influxdata / influxdata/influxdb

All data has been lost during upgrade from 2.7.3 to 2.7.4

Open
#24,467 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behaviour.

1. Install and set up influxdb 2.7.3.
2. Ingest some data to bucket(s).
3. Upgrade to influxdb 2.7.4.

__Expected behaviour:__
All settings and data retained.

__Actual behaviour:__
influxdb actually acted as a fresh installation, asking to proceed with initial bootstrap (create organization, user, default bucket etc.)

__Environment info:__

VM powered by Ubuntu 22.04:
```
root@xxx:/var/log# uname -a
Linux xxx 5.15.0-1047-oracle #53-Ubuntu SMP Wed Oct 18 09:56:06 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

root@xxx:/var/log# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
```

influxdb was installed from official repo:
```
root@xxx:/var/log# cat /etc/apt/sources.list.d/influxdata.list
deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg] https://repos.influxdata.com/debian stable main

root@xxx:/var/log# dpkg -s influxdb2
Package: influxdb2
Status: install ok installed
Priority: optional
Section: default
Installed-Size: 96975
Maintainer: support@influxdb.com
Architecture: arm64
Version: 2.7.4-1
Depends: curl
Recommends: influxdb2-cli
Conflicts: influxdb
Conffiles:
/etc/logrotate.d/influxdb 546afc8a077d25862a8c0a4c1220223f
Description: Distributed time-series database.
License: MIT
Vendor: InfluxData
Homepage: https://influxdata.com

root@xxx:/var/log# dpkg -s influxdb2-cli
Package: influxdb2-cli
Status: install ok installed
Priority: optional
Section: default
Installed-Size: 23530
Maintainer: support@influxdb.com
Architecture: arm64
Version: 2.7.3-1
Description: CLI for managing resources in InfluxDB v2
License: MIT
Vendor: InfluxData
Homepage: https://influxdata.com
```

dpkg logs (1st installation was on October 23rd):
```
2023-10-23 19:32:04 install influxdb2:arm64 2.7.3-1
2023-10-23 19:32:04 status half-installed influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 status unpacked influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 install influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 status half-installed influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 status unpacked influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 configure influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 status unpacked influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 status half-configured influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 status installed influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 configure influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 status unpacked influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 status half-configured influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 status installed influxdb2-cli:arm64 2.7.3-1
2023-11-17 13:13:06 upgrade influxdb2:arm64 2.7.3-1 2.7.4-1
2023-11-17 13:13:06 status half-configured influxdb2:arm64 2.7.3-1
2023-11-17 13:13:06 status unpacked influxdb2:arm64 2.7.3-1
2023-11-17 13:13:06 status half-installed influxdb2:arm64 2.7.3-1
2023-11-17 13:13:07 status unpacked influxdb2:arm64 2.7.4-1
2023-11-17 13:13:08 configure influxdb2:arm64 2.7.4-1
2023-11-17 13:13:08 status unpacked influxdb2:arm64 2.7.4-1
2023-11-17 13:13:08 status half-configured influxdb2:arm64 2.7.4-1
2023-11-17 13:13:08 status installed influxdb2:arm64 2.7.4-1
```

__Config:__
Nothing fancy in the config, pretty much everything was on default.

__Logs:__

Excerpt from syslog covering the upgrade time window:
```
Nov 17 13:07:50 xxx influxd-systemd-start.sh[859]: ts=2023-11-17T13:07:50.796846Z lvl=info msg="Retention policy deletion check (start)" log_id=0lLIe6W0000 service=retention op_name=retention_delete_check op_event=start
Nov 17 13:07:50 xxx influxd-systemd-start.sh[859]: ts=2023-11-17T13:07:50.805460Z lvl=info msg="Retention policy deletion check (end)" log_id=0lLIe6W0000 service=retention op_name=retention_delete_check op_event=end op_elapsed=9.435ms
Nov 17 13:13:23 xxx systemd[1]: Stopping InfluxDB is an open-source, distributed, time series database...
Nov 17 13:13:23 xxx systemd[1]: influxdb.service: Deactivated successfully.
Nov 17 13:13:23 xxx systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Nov 17 13:13:23 xxx systemd[1]: influxdb.service: Consumed 11min 17.299s CPU time.
Nov 17 13:13:23 xxx systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3081994]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:23.855404Z lvl=info msg="Welcome to InfluxDB" log_id=0lZMUCZ0000 version=v2.7.4 commit=19e5c0e1b7 build_date=2023-11-08T17:07:27Z log_level=info
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:23.878622Z lvl=info msg="Resources opened" log_id=0lZMUCZ0000 service=bolt path=/var/lib/influxdb/influxd.bolt
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:23.881165Z lvl=info msg="Resources opened" log_id=0lZMUCZ0000 service=sqlite path=/var/lib/influxdb/influxd.sqlite
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:23.886946Z lvl=info msg="Bringing up metadata migrations" log_id=0lZMUCZ0000 service="KV migrations" migration_count=20
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3082007]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3082017]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.103879Z lvl=info msg="Bringing up metadata migrations" log_id=0lZMUCZ0000 service="SQL migrations" migration_count=8
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081992]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.171118Z lvl=info msg="Using data dir" log_id=0lZMUCZ0000 service=storage-engine service=store path=/var/lib/influxdb/engine/data
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.171654Z lvl=info msg="Compaction settings" log_id=0lZMUCZ0000 service=storage-engine service=store max_concurrent_compactions=1 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.171818Z lvl=info msg="Open store (start)" log_id=0lZMUCZ0000 service=storage-engine service=store op_name=tsdb_open op_event=start
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.172213Z lvl=info msg="Open store (end)" log_id=0lZMUCZ0000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=0.397ms
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.172661Z lvl=info msg="Starting retention policy enforcement service" log_id=0lZMUCZ0000 service=retention check_interval=30m
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.172795Z lvl=info msg="Starting precreation service" log_id=0lZMUCZ0000 service=shard-precreation check_interval=10m advance_period=30m
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.178994Z lvl=info msg="Starting query controller" log_id=0lZMUCZ0000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.186139Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0lZMUCZ0000 max_select_point=0 max_select_series=0 max_select_buckets=0
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.215891Z lvl=info msg=Starting log_id=0lZMUCZ0000 service=telemetry interval=8h
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.217681Z lvl=info msg=Listening log_id=0lZMUCZ0000 service=tcp-listener transport=http addr=:8086 port=8086
Nov 17 13:13:25 xxx influxd-systemd-start.sh[3081992]: InfluxDB started
Nov 17 13:13:25 xxx systemd[1]: Started InfluxDB is an open-source, distributed, time series database.
Nov 17 13:13:47 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:47.855330Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
Nov 17 13:13:48 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:48.347066Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="session not found"
Nov 17 13:13:57 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:57.519724Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
Nov 17 13:14:07 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:14:07.521025Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
Nov 17 13:14:17 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:14:17.521916Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
Nov 17 13:14:27 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:14:27.527104Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
Nov 17 13:14:37 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:14:37.523561Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
```

_May_ be related to https://github.com/influxdata/influxdb/issues/23849 ?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Ubuntu package upgrade from influxdb2 2.7.3-1 to 2.7.4-1 with existing data, then inspect the reported systemd logs around metadata migrations and the configured paths /var/lib/influxdb/influxd.bolt, /var/lib/influxdb/influxd.sqlite, and /var/lib/influxdb/engine/data. Done means an upgrade retains the existing bootstrap state, settings, and data instead of starting fresh.

Written by the indexing model from the issue text.

Assessment

Tech stack
ubuntu
Domain
databases, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.