influxdata / influxdata/influxdb
Backing up fails due to a shard error
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Populate an influxdb instance
3. Run influx backup
3. ...
__Expected behavior:__
Backup completes by downloading data from all shards
__Actual behavior:__
Downloading the backup tar.gz files error out periodically when trying to download a specific shard. The behavior seems to be non determinustic
__Environment info:__
* System info: Linux 3.10.0-1160.25.1.el7.x86_64 x86_64
* InfluxDB version: InfluxDB 2.0.6 (git: 4db98b4c9a) build_date: 2021-04-29T16:48:12Z
* Other relevant environment details: None
__Config:__
Copy any non-default config values here or attach the full config as a gist or file.
__Logs:__
2021-06-24T22:15:48.990826Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 699, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s699.tar.gz"}
2021-06-24T22:15:48.994955Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 740, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s740.tar.gz"}
2021-06-24T22:15:49.000888Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 781, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s781.tar.gz"}
2021-06-24T22:15:49.004757Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 822, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s822.tar.gz"}
2021-06-24T22:15:49.008300Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 863, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s863.tar.gz"}
2021-06-24T22:15:49.012714Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 906, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s906.tar.gz"}
2021-06-24T22:15:49.016365Z info Backing up shard {"log_id": "0Ux1LLx0000", "id": 947, "path": "backup.dir/2021-06-24_test/20210624T221548Z.s947.tar.gz"}
Error: Failed to download shard backup: An internal error has occurred.
__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.
```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for ~30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=30s"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz` and `iostat.txt` output files.
```
This is possibly a duplicate of #16739. But this bug is for influxd 2.0
Contributor guide
Research direction
Start by reproducing the failure with the `influx backup ` command against a populated InfluxDB 2.0.6 instance, using the shard-backup log entries and reported internal error as the observed behavior. Compare the issue with possible duplicate #16739; done means the backup completes and downloads data from all shards without intermittent shard errors.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100