influxdata / influxdata/influxdb
Backups created on one instance cannot be restored to another: "An internal error has occurred"
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
This issue is raised from a [conversation on the InfluxDB Community Slack ](https://influxcommunity.slack.com/archives/CH8RV8PK5/p1660072746143279)with @samhld. I've been trying to transfer bucket data between InfluxDB instances using the backup/restore functionality and seeing problems with the restore operation.
The influx restore call appears to fail:
```
❯ influx restore --full . -t wJMugsVT2fp8MBiFqqPcOVO2yYvUSdAAU0Ou9xbbf7RTJ293ewGMJqOjeAs9EP6edth-8C1b_ssoPNgGQ8s27g==
2022-07-28T17:30:43.318507Z info Restoring full metadata from local backup {"log_id": "0byTbrxG000", "path": "20220728T003806Z.bolt.gz"}
Error: Failed to upload local KV backup at "20220728T003806Z.bolt.gz": An internal error has occurred - check server logs.
See 'influx restore -h' for help
```
The server logs then show a slightly less confusing message:
```
❯ sudo journalctl -fu influxdb
-- Logs begin at Sun 2022-07-10 19:19:14 PDT. --
Jul 28 08:55:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T15:55:10.151870Z lvl=info msg="Retention policy deletion check (start)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=start
Jul 28 08:55:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T15:55:10.152082Z lvl=info msg="Retention policy deletion check (end)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=end op_elapsed=0.232ms
Jul 28 09:25:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T16:25:10.151421Z lvl=info msg="Retention policy deletion check (start)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=start
Jul 28 09:25:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T16:25:10.151927Z lvl=info msg="Retention policy deletion check (end)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=end op_elapsed=0.516ms
Jul 28 09:55:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T16:55:10.151855Z lvl=info msg="Retention policy deletion check (start)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=start
Jul 28 09:55:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T16:55:10.152669Z lvl=info msg="Retention policy deletion check (end)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=end op_elapsed=0.833ms
Jul 28 10:25:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T17:25:10.152179Z lvl=info msg="Retention policy deletion check (start)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=start
Jul 28 10:25:10 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T17:25:10.152724Z lvl=info msg="Retention policy deletion check (end)" log_id=0bxTmYe0000 service=retention op_name=retention_delete_check op_event=end op_elapsed=0.566ms
Jul 28 10:25:45 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T17:25:45.734845Z lvl=warn msg="internal error not returned to client" log_id=0bxTmYe0000 handler=error_logger error="unable to open boltdb: invalid database"
Jul 28 10:30:43 hitl-nuc influxd-systemd-start.sh[11661]: ts=2022-07-28T17:30:43.388630Z lvl=warn msg="internal error not returned to client" log_id=0bxTmYe0000 handler=error_logger error="unable to open boltdb: invalid database"
```
AFAIK I'm running the latest versions of InfluxDB and Influx CLI:
```
❯ influxd version
InfluxDB v2.3.0+SNAPSHOT.090f681737 (git: 090f681737) build_date: 2022-06-16T19:33:50Z
❯ influx version
Influx CLI 2.3.0 (git: 88ba346) build_date: 2022-04-06T19:30:53Z
```
@samhld indicated that he has escalated this to the engineering team and suggested that I file a report here for further investigation. Please let me know what additional information would be helpful in the debugging process.
Contributor guide
Research direction
Start by reproducing the documented `influx restore --full` command with a backup created on another instance, then compare the CLI error with the server log message `unable to open boltdb: invalid database`. The issue is done when cross-instance restore succeeds without the internal error and the relevant failure is reported clearly.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100