influxdata / influxdata/influxdb

Backing up fails due to a shard error

Open
#21,743 4 comments 0 reactions 0 assignees View on GitHub
area/2.x area/backup and restore kind/bug
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. Populate an influxdb instance
3. Run influx backup
3. ...

__Expected behavior:__
Backup completes by downloading data from all shards

__Actual behavior:__
Downloading the backup tar.gz files error out periodically when trying to download a specific shard. The behavior seems to be non determinustic

__Environment info:__

* System info: Linux 3.10.0-1160.25.1.el7.x86_64 x86_64
* InfluxDB version: InfluxDB 2.0.6 (git: 4db98b4c9a) build_date: 2021-04-29T16:48:12Z
* Other relevant environment details: None

__Config:__
Copy any non-default config values here or attach the full config as a gist or file.

__Logs:__
2021-06-24T22:15:48.990826Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 699, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s699.tar.gz"}
2021-06-24T22:15:48.994955Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 740, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s740.tar.gz"}
2021-06-24T22:15:49.000888Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 781, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s781.tar.gz"}
2021-06-24T22:15:49.004757Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 822, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s822.tar.gz"}
2021-06-24T22:15:49.008300Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 863, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s863.tar.gz"}
2021-06-24T22:15:49.012714Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 906, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s906.tar.gz"}
2021-06-24T22:15:49.016365Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 947, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s947.tar.gz"}
Error: Failed to download shard backup: An internal error has occurred.

__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.

```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for ~30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=30s"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz` and `iostat.txt` output files.
```

This is possibly a duplicate of #16739. But this bug is for influxd 2.0

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with the `influx backup ` command against a populated InfluxDB 2.0.6 instance, using the shard-backup log entries and reported internal error as the observed behavior. Compare the issue with possible duplicate #16739; done means the backup completes and downloads data from all shards without intermittent shard errors.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.