influxdata / influxdata/influxdb

Unreliable backup command

Open
#25,592 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

The CLI `backup` command is not reliable and does not provide a usable UI, for the following reasons:

1. It does not implement commonly used output configuration options that would be suitable for backup scripts, i.e. it only has a single output mode: **everything**. This makes it impossible for the administrator to verify a backup's completeness and integrity as they would have to read the (lengthy) output for each backup created. For example, there could be a "quiet-mode" that only creates output on warnings or errors, which could be used in backup scripts and cronjobs to have the admin notified if something goes wrong.
2. Backing up frequently creates **a lot** of data as each snapshot is chunked independently and common data of subsequent snapshots cannot be reused. In instances with many or large buckets, this fills storage quite quickly and forces admins to only keep a low number of snapshots, where effective data is heavily redundant. This even may go to only keeping a single snapshot and removing it prior to the next backup and if this goes wrong and the instance fails, all data is lost.
3. InfluxDB is not entering a backup mode where it would lock bucket access during a backup. Thus it is still accepting read and write requests during the backup (as well during a restore). This is something many databases have implemented for a reason: to prevent inconsistent data being backed up. To prevent this, an admin would have to ensure that all access to the instance is blocked during backup/restore, which can be quite a task when the instance is usually accessible in a large local network and not all access is piped through a single instance such as a web server.
4. In our instance (with a data dir of ~50GB) the backup is really slow (see logs below).

The specific issue that we ran into: After a server reboot, our instance was unresponsive with no clue in the logs, what was wrong. When trying to test restore to a blank instance (which we had tested earlier this year) we saw that about 50% of the weekly saved snapshot were corrupt (data chunks in `.tar.gz` as well as boltDB and SQLite dumps were present, manifest files were somehow missing). The backup outputs, piped from the server backup script to a log file, only showed an error message that the InfluxDB API could not be reached -- we had forgotten to pipe the output in append mode and rotate the log file. The youngest consistent weekly was 8 weeks old. As the instance could not be repaired with reasonable amount of work, we decided to restore and settle with the loss of 8 weeks of data.

__Environment info:__

* InfluxDB version: InfluxDB v2.7.10 (git: f302d9730c) build_date: 2024-08-16T20:19:28Z
* InfluxDB CLI version: Influx CLI dev (git: a79a2a1b825867421d320428538f76a4c90aa34c) build_date: 2024-04-16T14:34:32Z
* System info: `Linux 5.15.0-126-generic x86_64`

__Logs:__

This is the routine output a recent backup:

```
2024/11/23 02:30:03 INFO: Downloading metadata snapshot
2024/11/23 02:30:03 INFO: Backing up TSM for shard 5031
2024/11/23 02:30:03 INFO: Backing up TSM for shard 5032
2024/11/23 02:30:03 INFO: Backing up TSM for shard 4980
[This goes on...]
2024/11/23 03:26:55 INFO: Backing up TSM for shard 6829
2024/11/23 03:26:56 INFO: Backing up TSM for shard 6886
2024/11/23 03:27:11 INFO: Backing up TSM for shard 6887
2024/11/23 03:29:32 INFO: Backing up TSM for shard 6952
```

Contributor guide

Open the contributing guide

Research direction

The report names the CLI backup command, backup and restore behavior, snapshot chunking, and the logged output, but no source files or tests. Start by narrowing the four concerns to one reproducible failure, using the provided logs and restore-to-blank-instance scenario. Done should be a specific agreed behavior with reproducible validation for that scope.

Written by the indexing model from the issue text.

Assessment

Domain
cli, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.