influxdata / influxdata/influxdb
influxdb2 won't start because of missing metadata for bucket
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Hello,
I have a influxdb2 server which was running fine for weeks. Yesterday, I had to restart the db. After restarting, the influxdb failed to start with the following error:
```
Oct 14 14:06:10 telemetry01 influxd-systemd-start.sh[1914162]: ts=2024-10-14T14:06:10.797458Z lvl=error msg="Missing metadata for bucket." log_id=0sEqSkP0000 bucket=_monitoring bucket_id=dce6500dbda1fbce
Oct 14 14:06:10 telemetry01 influxd-systemd-start.sh[1914162]: ts=2024-10-14T14:06:10.797468Z lvl=error msg="Incompatible InfluxDB 2.0 metadata found. File must be moved before influxd will start." log_id=0sEqSkP0000 path=/var/lib/influxdb/influxd.bolt
Oct 14 14:06:10 telemetry01 influxd-systemd-start.sh[1914162]: ts=2024-10-14T14:06:10.797831Z lvl=error msg="Incompatible InfluxDB 2.0 version found. Move all files outside of engine_path before influxd will start." log_id=0sEqSkP0000 engine_path=/var/lib/influxdb/engine
```
Please assist me with repairing my setup.
I have a backup of my bolt file and `/var/lib/influxdb/` folder
__Steps to reproduce:__
List the minimal actions needed to reproduce the behaviour.
1. systemctl start influxdb.service
__Expected behaviour:__
influxdb should start up fine.
__Actual behaviour:__
After the influxdb failed to start, I used boltbrowser to edit my `/var/lib/influxdb/influxd.bolt` File.
I deleted the bucket "_monitoring" with the id "dce6500dbda1fbce" from "bucketsv1" as shown in the initial error message. After that, the influxdb was starting again but all my buckets are no longer accessible from both, cli and WebUI:
Output of: `$ influx bucket ls`:
`Error: failed to list buckets: 404 Not Found: bucket not found`
Server Log:
`2024-10-15T09:10:59.940222Z error api error encountered {"log_id": "0sFrxh5l000", "handler": "bucket", "error": "bucket not found"}`
The same error occurs, if I try to open the Data Explorer dashboard in the WebUI.
I can still see the token of my user:
```
$ influx auth list|head -n 2
ID Description Token User Name User ID Permissions
0d12c7f2307c2000 influxadm's Token influxadm 0d12c7f21abc2000 [read:/authorizations write:/authorizations read:/buckets write:/buckets read:/dashboards write:/dashboards read:/orgs write:/orgs read:/sources write:/sources read:/tasks write:/tasks read:/telegrafs write:/telegrafs read:/users write:/users read:/variables write:/variables read:/scrapers write:/scrapers read:/secrets write:/secrets read:/labels write:/labels read:/views write:/views read:/documents write:/documents read:/notificationRules write:/notificationRules read:/notificationEndpoints write:/notificationEndpoints read:/checks write:/checks read:/dbrp write:/dbrp read:/notebooks write:/notebooks read:/annotations write:/annotations read:/remotes write:/remotes read:/replications write:/replications]
```
But if I use this token to check the buckets of the org, I run into the following error:
```
$ influx bucket list -o -t
Error: failed to list buckets: 401 Unauthorized: unauthorized access
```
Server Log:
`2024-10-15T09:17:52.171981Z info Unauthorized {"log_id": "0sFrxh5l000", "error": "authorization not found"}`
If I try to create a bucket with the name of a bucket that existed before, this happens:
```
$ influx bucket create -o -n
Error: failed to create bucket: 422 Unprocessable Entity: bucket with name already exists
```
So to me it seems that the buckets are not really gone but the index is corrupted?
__Environment info:__
Ubuntu 24.04 LTS on Virtual Machine
influxdb2 with version 2.7.6-1 from official repos
16GB RAM / 4 Cores
__Config:__
default
__Logs:__
see above
Contributor guide
Research direction
Start with the reported startup logs and the backed-up /var/lib/influxdb/ directory, especially influxd.bolt and the engine path. Reproduce the failure with systemctl start influxdb.service and inspect the bucketsv1 metadata alongside the documented 2.7.6 behavior. Done means the server starts without metadata errors and the existing buckets, authorization token, CLI, and WebUI work again.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100