influxdata / influxdata/influxdb
Influx 2.0.5 restore fails with: Error: failed to restore KV snapshot: InfluxDB OSS-only command failed: 500 Internal Server Error: An internal error has occurred
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Background:__
I'm trying to upgrade from 2.0.5 to 2.1.1 on a separate raspberry pi. I've tried to restore the 2.0.5 backup both to a fresh install of 2.1.1, and when that failed, a fresh install of 2.0.5.
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. On a populated 2.0.5 database run "influx backup name_of_backup_dir -t $TOKEN"
2. Copy directory to a new raspberry pi and follow install instructions:
3. wget https://dl.influxdata.com/influxdb/releases/influxdb2-2.0.5-linux-arm64.tar.gz (Ideally, I'd want 2.1.1 to work here)
4. tar xvzf influxdb2-2.0.5-linux-arm64.tar.gz
5. sudo cp influxdb2-2.0.5-linux-arm64/influxd /usr/local/bin/
6. wget https://dl.influxdata.com/influxdb/releases/influxdb2-client-2.2.0-linux-arm64.tar.gz
7. tar xvzf influxdb2-client-2.2.0-linux-arm64.tar.gz
8. sudo cp influxdb2-client-2.2.0-linux-arm64/influx /usr/local/bin/
9. influxd --reporting-disabled --storage-retention-check-interval=24h
10. influx setup --token $TOKEN
11. influx restore ~/backup name_of_backup_dir/ --full
__Expected behavior:__
All data restored and replaced with time series data and InfluxDB key-value data such as tokens, users, dashboards, from the original backup.
__Actual behavior:__
INFO: Restoring KV snapshot
Error: failed to restore KV snapshot: InfluxDB OSS-only command failed: 500 Internal Server Error: An internal error has occurred
Trying to view the web interface just shows a loading circle.
__Environment info:__
* System info: Linux 5.10.103-v8+ aarch64 GNU/Linux (Bullseye)
* InfluxDB version: InfluxDB 2.0.5 (git: 741389781e) build_date: 2021-04-27T17:57:07Z
* Other relevant environment details: Raspberry Pi3 with fresh OS, updated & upgraded, no other modifications made/programs installed.
__Config:__
Just the two command line flags: --reporting-disabled --storage-retention-check-interval=24h
Maybe important: the original backup database location was on a mounted SSD storage, whereas I'm restoring to a default but different) location.
__Logs:__
The 2.0.5 logs didn't give further details. At the risk on confusing the issue, the output from when I tried the same steps but with 2.1.1 were:
`2022-03-20T10:51:35.802221Z warn internal error not returned to client {"log_id": "0_LjC2sG000", "handler": "error_logger", "error": "[shard 606] optogen/606/index/3/MANIFEST: too many open files"}
2022-03-20T10:51:35.802309Z error manifest write failed, index is potentially damaged {"log_id": "0_LjC2sG000", "service": "storage-engine", "indexi/.influxdbv2/engine/data/14a8ce509a563be6/autogen/606/index/0/MANIFEST: too many open files"}
2022-03-20T10:51:35.802334Z error manifest write failed, index is potentially damaged {"log_id": "0_LjC2sG000", "service": "storage-engine", "indexi/.influxdbv2/engine/data/14a8ce509a563be6/autogen/606/index/2/MANIFEST: too many open files"}
Error: failed to restore KV snapshot: InfluxDB OSS-only command failed: 500 Internal Server Error: An internal error has occurred - check server logs`
Contributor guide
Research direction
Start with the reported `influx backup` and `influx restore --full` commands and compare behavior on InfluxDB 2.0.5 and 2.1.1. Check the server logs alongside the reported `too many open files` messages and determine the conditions causing the KV snapshot restore to return HTTP 500; done means the backup restores successfully or the failure is documented with a reproducible cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, raspberry-pi
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100