influxdata / influxdata/influxdb
All data has been lost during upgrade from 2.7.3 to 2.7.4
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behaviour.
1. Install and set up influxdb 2.7.3.
2. Ingest some data to bucket(s).
3. Upgrade to influxdb 2.7.4.
__Expected behaviour:__
All settings and data retained.
__Actual behaviour:__
influxdb actually acted as a fresh installation, asking to proceed with initial bootstrap (create organization, user, default bucket etc.)
__Environment info:__
VM powered by Ubuntu 22.04:
```
root@xxx:/var/log# uname -a
Linux xxx 5.15.0-1047-oracle #53-Ubuntu SMP Wed Oct 18 09:56:06 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
root@xxx:/var/log# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
```
influxdb was installed from official repo:
```
root@xxx:/var/log# cat /etc/apt/sources.list.d/influxdata.list
deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg] https://repos.influxdata.com/debian stable main
root@xxx:/var/log# dpkg -s influxdb2
Package: influxdb2
Status: install ok installed
Priority: optional
Section: default
Installed-Size: 96975
Maintainer: support@influxdb.com
Architecture: arm64
Version: 2.7.4-1
Depends: curl
Recommends: influxdb2-cli
Conflicts: influxdb
Conffiles:
/etc/logrotate.d/influxdb 546afc8a077d25862a8c0a4c1220223f
Description: Distributed time-series database.
License: MIT
Vendor: InfluxData
Homepage: https://influxdata.com
root@xxx:/var/log# dpkg -s influxdb2-cli
Package: influxdb2-cli
Status: install ok installed
Priority: optional
Section: default
Installed-Size: 23530
Maintainer: support@influxdb.com
Architecture: arm64
Version: 2.7.3-1
Description: CLI for managing resources in InfluxDB v2
License: MIT
Vendor: InfluxData
Homepage: https://influxdata.com
```
dpkg logs (1st installation was on October 23rd):
```
2023-10-23 19:32:04 install influxdb2:arm64 2.7.3-1
2023-10-23 19:32:04 status half-installed influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 status unpacked influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 install influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 status half-installed influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 status unpacked influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 configure influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 status unpacked influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 status half-configured influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 status installed influxdb2:arm64 2.7.3-1
2023-10-23 19:32:06 configure influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 status unpacked influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 status half-configured influxdb2-cli:arm64 2.7.3-1
2023-10-23 19:32:06 status installed influxdb2-cli:arm64 2.7.3-1
2023-11-17 13:13:06 upgrade influxdb2:arm64 2.7.3-1 2.7.4-1
2023-11-17 13:13:06 status half-configured influxdb2:arm64 2.7.3-1
2023-11-17 13:13:06 status unpacked influxdb2:arm64 2.7.3-1
2023-11-17 13:13:06 status half-installed influxdb2:arm64 2.7.3-1
2023-11-17 13:13:07 status unpacked influxdb2:arm64 2.7.4-1
2023-11-17 13:13:08 configure influxdb2:arm64 2.7.4-1
2023-11-17 13:13:08 status unpacked influxdb2:arm64 2.7.4-1
2023-11-17 13:13:08 status half-configured influxdb2:arm64 2.7.4-1
2023-11-17 13:13:08 status installed influxdb2:arm64 2.7.4-1
```
__Config:__
Nothing fancy in the config, pretty much everything was on default.
__Logs:__
Excerpt from syslog covering the upgrade time window:
```
Nov 17 13:07:50 xxx influxd-systemd-start.sh[859]: ts=2023-11-17T13:07:50.796846Z lvl=info msg="Retention policy deletion check (start)" log_id=0lLIe6W0000 service=retention op_name=retention_delete_check op_event=start
Nov 17 13:07:50 xxx influxd-systemd-start.sh[859]: ts=2023-11-17T13:07:50.805460Z lvl=info msg="Retention policy deletion check (end)" log_id=0lLIe6W0000 service=retention op_name=retention_delete_check op_event=end op_elapsed=9.435ms
Nov 17 13:13:23 xxx systemd[1]: Stopping InfluxDB is an open-source, distributed, time series database...
Nov 17 13:13:23 xxx systemd[1]: influxdb.service: Deactivated successfully.
Nov 17 13:13:23 xxx systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Nov 17 13:13:23 xxx systemd[1]: influxdb.service: Consumed 11min 17.299s CPU time.
Nov 17 13:13:23 xxx systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3081994]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:23.855404Z lvl=info msg="Welcome to InfluxDB" log_id=0lZMUCZ0000 version=v2.7.4 commit=19e5c0e1b7 build_date=2023-11-08T17:07:27Z log_level=info
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:23.878622Z lvl=info msg="Resources opened" log_id=0lZMUCZ0000 service=bolt path=/var/lib/influxdb/influxd.bolt
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:23.881165Z lvl=info msg="Resources opened" log_id=0lZMUCZ0000 service=sqlite path=/var/lib/influxdb/influxd.sqlite
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:23.886946Z lvl=info msg="Bringing up metadata migrations" log_id=0lZMUCZ0000 service="KV migrations" migration_count=20
Nov 17 13:13:23 xxx influxd-systemd-start.sh[3082007]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3082017]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.103879Z lvl=info msg="Bringing up metadata migrations" log_id=0lZMUCZ0000 service="SQL migrations" migration_count=8
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081992]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.171118Z lvl=info msg="Using data dir" log_id=0lZMUCZ0000 service=storage-engine service=store path=/var/lib/influxdb/engine/data
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.171654Z lvl=info msg="Compaction settings" log_id=0lZMUCZ0000 service=storage-engine service=store max_concurrent_compactions=1 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.171818Z lvl=info msg="Open store (start)" log_id=0lZMUCZ0000 service=storage-engine service=store op_name=tsdb_open op_event=start
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.172213Z lvl=info msg="Open store (end)" log_id=0lZMUCZ0000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=0.397ms
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.172661Z lvl=info msg="Starting retention policy enforcement service" log_id=0lZMUCZ0000 service=retention check_interval=30m
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.172795Z lvl=info msg="Starting precreation service" log_id=0lZMUCZ0000 service=shard-precreation check_interval=10m advance_period=30m
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.178994Z lvl=info msg="Starting query controller" log_id=0lZMUCZ0000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.186139Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0lZMUCZ0000 max_select_point=0 max_select_series=0 max_select_buckets=0
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.215891Z lvl=info msg=Starting log_id=0lZMUCZ0000 service=telemetry interval=8h
Nov 17 13:13:24 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:24.217681Z lvl=info msg=Listening log_id=0lZMUCZ0000 service=tcp-listener transport=http addr=:8086 port=8086
Nov 17 13:13:25 xxx influxd-systemd-start.sh[3081992]: InfluxDB started
Nov 17 13:13:25 xxx systemd[1]: Started InfluxDB is an open-source, distributed, time series database.
Nov 17 13:13:47 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:47.855330Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
Nov 17 13:13:48 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:48.347066Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="session not found"
Nov 17 13:13:57 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:13:57.519724Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
Nov 17 13:14:07 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:14:07.521025Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
Nov 17 13:14:17 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:14:17.521916Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
Nov 17 13:14:27 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:14:27.527104Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
Nov 17 13:14:37 xxx influxd-systemd-start.sh[3081993]: ts=2023-11-17T13:14:37.523561Z lvl=info msg=Unauthorized log_id=0lZMUCZ0000 error="authorization not found"
```
_May_ be related to https://github.com/influxdata/influxdb/issues/23849 ?
Contributor guide
Research direction
Start by reproducing the Ubuntu package upgrade from influxdb2 2.7.3-1 to 2.7.4-1 with existing data, then inspect the reported systemd logs around metadata migrations and the configured paths /var/lib/influxdb/influxd.bolt, /var/lib/influxdb/influxd.sqlite, and /var/lib/influxdb/engine/data. Done means an upgrade retains the existing bootstrap state, settings, and data instead of starting fresh.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ubuntu
- Domain
- databases, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100