influxdata / influxdata/influxdb

Backups created on one instance cannot be restored to another: "An internal error has occurred"

Open
#23,619 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

This issue is raised from a [conversation on the InfluxDB Community Slack ](https://influxcommunity.slack.com/archives/CH8RV8PK5/p1660072746143279)with @samhld. I've been trying to transfer bucket data between InfluxDB instances using the backup/restore functionality and seeing problems with the restore operation.

The influx restore call appears to fail:

```
❯ influx restore --full . -t wJMugsVT2fp8MBiFqqPcOVO2yYvUSdAAU0Ou9xbbf7RTJ293ewGMJqOjeAs9EP6edth-8C1b_ssoPNgGQ8s27g==
2022-07-28T17:30:43.318507Z info Restoring full metadata from local backup {"log_id": "0byTbrxG000", "path": "20220728T003806Z.bolt.gz"}
Error: Failed to upload local KV backup at "20220728T003806Z.bolt.gz": An internal error has occurred - check server logs.
See 'influx restore -h' for help
```

The server logs then show a slightly less confusing message:

```
❯ sudo journalctl -fu influxdb
-- Logs begin at Sun 2022-07-10 19:19:14 PDT. --
Jul 28 08:55:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T15:55:10.151870Z lvl=info msg="Retention policy deletion check (start)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=start
Jul 28 08:55:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T15:55:10.152082Z lvl=info msg="Retention policy deletion check (end)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=end op_elapsed=0.232ms
Jul 28 09:25:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T16:25:10.151421Z lvl=info msg="Retention policy deletion check (start)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=start
Jul 28 09:25:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T16:25:10.151927Z lvl=info msg="Retention policy deletion check (end)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=end op_elapsed=0.516ms
Jul 28 09:55:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T16:55:10.151855Z lvl=info msg="Retention policy deletion check (start)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=start
Jul 28 09:55:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T16:55:10.152669Z lvl=info msg="Retention policy deletion check (end)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=end op_elapsed=0.833ms
Jul 28 10:25:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T17:25:10.152179Z lvl=info msg="Retention policy deletion check (start)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=start
Jul 28 10:25:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T17:25:10.152724Z lvl=info msg="Retention policy deletion check (end)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=end op_elapsed=0.566ms
Jul 28 10:25:45 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T17:25:45.734845Z lvl=warn msg="internal error not returned to client" log_id=0bxTmYe0000 handler=error_logger error="unable to open boltdb: invalid database"
Jul 28 10:30:43 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T17:30:43.388630Z lvl=warn msg="internal error not returned to client" log_id=0bxTmYe0000 handler=error_logger error="unable to open boltdb: invalid database"
```

AFAIK I'm running the latest versions of InfluxDB and Influx CLI:

```
❯ influxd version
InfluxDB v2.3.0+SNAPSHOT.090f681737 (git: 090f681737) build_date: 2022-06-16T19:33:50Z
❯ influx version
Influx CLI 2.3.0 (git: 88ba346) build_date: 2022-04-06T19:30:53Z
```

@samhld indicated that he has escalated this to the engineering team and suggested that I file a report here for further investigation. Please let me know what additional information would be helpful in the debugging process.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the documented `influx restore --full` command with a backup created on another instance, then compare the CLI error with the server log message `unable to open boltdb: invalid database`. The issue is done when cross-instance restore succeeds without the internal error and the relevant failure is reported clearly.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.