influxdata / influxdata/influxdb

Influx 2.0.5 restore fails with: Error: failed to restore KV snapshot: InfluxDB OSS-only command failed: 500 Internal Server Error: An internal error has occurred

Open
#23,212 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Background:__
I'm trying to upgrade from 2.0.5 to 2.1.1 on a separate raspberry pi. I've tried to restore the 2.0.5 backup both to a fresh install of 2.1.1, and when that failed, a fresh install of 2.0.5.

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. On a populated 2.0.5 database run "influx backup name_of_backup_dir -t $TOKEN"
2. Copy directory to a new raspberry pi and follow install instructions:
3. wget https://dl.influxdata.com/influxdb/releases/influxdb2-2.0.5-linux-arm64.tar.gz (Ideally, I'd want 2.1.1 to work here)
4. tar xvzf influxdb2-2.0.5-linux-arm64.tar.gz
5. sudo cp influxdb2-2.0.5-linux-arm64/influxd /usr/local/bin/
6. wget https://dl.influxdata.com/influxdb/releases/influxdb2-client-2.2.0-linux-arm64.tar.gz
7. tar xvzf influxdb2-client-2.2.0-linux-arm64.tar.gz
8. sudo cp influxdb2-client-2.2.0-linux-arm64/influx /usr/local/bin/
9. influxd --reporting-disabled --storage-retention-check-interval=24h
10. influx setup --token $TOKEN
11. influx restore ~/backup name_of_backup_dir/ --full

__Expected behavior:__
All data restored and replaced with time series data and InfluxDB key-value data such as tokens, users, dashboards, from the original backup.

__Actual behavior:__
INFO: Restoring KV snapshot
Error: failed to restore KV snapshot: InfluxDB OSS-only command failed: 500 Internal Server Error: An internal error has occurred

Trying to view the web interface just shows a loading circle.

__Environment info:__
* System info: Linux 5.10.103-v8+ aarch64 GNU/Linux (Bullseye)
* InfluxDB version: InfluxDB 2.0.5 (git: 741389781e) build_date: 2021-04-27T17:57:07Z
* Other relevant environment details: Raspberry Pi3 with fresh OS, updated & upgraded, no other modifications made/programs installed.

__Config:__
Just the two command line flags: --reporting-disabled --storage-retention-check-interval=24h
Maybe important: the original backup database location was on a mounted SSD storage, whereas I'm restoring to a default but different) location.

__Logs:__
The 2.0.5 logs didn't give further details. At the risk on confusing the issue, the output from when I tried the same steps but with 2.1.1 were:
`2022-03-20T10:51:35.802221Z warn internal error not returned to client {"log_id": "0_LjC2sG000", "handler": "error_logger", "error": "[shard 606] optogen/606/index/3/MANIFEST: too many open files"}
2022-03-20T10:51:35.802309Z error manifest write failed, index is potentially damaged {"log_id": "0_LjC2sG000", "service": "storage-engine", "indexi/.influxdbv2/engine/data/14a8ce509a563be6/autogen/606/index/0/MANIFEST: too many open files"}
2022-03-20T10:51:35.802334Z error manifest write failed, index is potentially damaged {"log_id": "0_LjC2sG000", "service": "storage-engine", "indexi/.influxdbv2/engine/data/14a8ce509a563be6/autogen/606/index/2/MANIFEST: too many open files"}
Error: failed to restore KV snapshot: InfluxDB OSS-only command failed: 500 Internal Server Error: An internal error has occurred - check server logs`

Contributor guide

Open the contributing guide

Research direction

Start with the reported `influx backup` and `influx restore --full` commands and compare behavior on InfluxDB 2.0.5 and 2.1.1. Check the server logs alongside the reported `too many open files` messages and determine the conditions causing the KV snapshot restore to return HTTP 500; done means the backup restores successfully or the failure is documented with a reproducible cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, raspberry-pi
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.