influxdata / influxdata/influxdb
influxd-ctl update-data on nodes with >20 GB data on disk
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
We currently recommend that folks use [`influxd-ctl update-data`](https://docs.influxdata.com/enterprise_influxdb/v1.7/administration/cluster-commands/#update-data) to update a data node’s address in the meta store, for example, when replacing a data node in a cluster.
The Support team has found that it's **not advisable** to use `influxd-ctl update-data`for nodes with **more than about 20 GB of data on disk**. The following workaround is being used:
1. influxd-ctl add-data
2. Manually rebalance shards, reaffirming RF2 is in place at every step
3. influxd-ctl copy-shard to copy shards to additional node
4. influxd-ctl remove-shard to remove shards from old node
5. Remove data node: influxd-ctl remove-data
**We're looking for help on the following:**
- Verify whether more than 20 GB of data on disk is causing issues when using `influxd-ctl update-data`
- Identify other possible variables causing this issue
- Determine next steps and best practice all things considered (in the interim and long term solution).
__Expected behavior:__
Be able to use [`influxd-ctl update-data`] for nodes with more than 20 GB of data on disk
Asking @abalone23 and @andrew-influx if they can provide more details below.
__Actual behavior:__
Describe What actually happened.
__Environment info:__
* System info: Run `uname -srm` and copy the output here
* InfluxDB version: Run `influxd version` and copy the output here
* Other relevant environment details: Container runtime, disk info, etc
__Config:__
Copy any non-default config values here or attach the full config as a gist or file.
__Logs:__
Include snippet of errors in log.
__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.
```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for at least 30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"
curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz`, `vars.txt`, and `iostat.txt` output files.
```
Contributor guide
Assessment
This issue has not been assessed yet.