influxdata / influxdata/influxdb

influxd-ctl update-data on nodes with >20 GB data on disk

Open
#17,592 1 comment 0 reactions 1 assignee Claimed by @mkm-influx View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

We currently recommend that folks use [`influxd-ctl update-data`](https://docs.influxdata.com/enterprise_influxdb/v1.7/administration/cluster-commands/#update-data) to update a data node’s address in the meta store, for example, when replacing a data node in a cluster.

The Support team has found that it's **not advisable** to use `influxd-ctl update-data`for nodes with **more than about 20 GB of data on disk**. The following workaround is being used:

1. influxd-ctl add-data
2. Manually rebalance shards, reaffirming RF2 is in place at every step
3. influxd-ctl copy-shard to copy shards to additional node
4. influxd-ctl remove-shard to remove shards from old node
5. Remove data node: influxd-ctl remove-data

**We're looking for help on the following:**
- Verify whether more than 20 GB of data on disk is causing issues when using `influxd-ctl update-data`
- Identify other possible variables causing this issue
- Determine next steps and best practice all things considered (in the interim and long term solution).

__Expected behavior:__
Be able to use [`influxd-ctl update-data`] for nodes with more than 20 GB of data on disk

Asking @abalone23 and @andrew-influx if they can provide more details below.
__Actual behavior:__
Describe What actually happened.

__Environment info:__

* System info: Run `uname -srm` and copy the output here
* InfluxDB version: Run `influxd version` and copy the output here
* Other relevant environment details: Container runtime, disk info, etc

__Config:__
Copy any non-default config values here or attach the full config as a gist or file.

__Logs:__
Include snippet of errors in log.

__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.

```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for at least 30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"
curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz`, `vars.txt`, and `iostat.txt` output files.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.