influxdata / influxdata/influxdb

Ensure influxd migrate tool doesn't migrate hot shards by default

Open
#17,250 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/storage area/tooling difficulty/low
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

The desirable behaviour for the influxd migrate tool is that it won't import hot shards. Importing hot shards can leave data behind inside of the in-memory cache and WAL files.

The tool has a flag that let's the operator specify they are OK with migrating hot shards (--migrate-hot-shards), but currently the tool does not actually check if a shard is hot.

We have APIs to check if a shard is hot, e.g., IsIdle() in the Engine code, but we can't use that when influxd is not running. Instead I propose that we simply check for none-truncated WAL file segments in the shard. If we detect any, and the user has not specified --migrate-hot-shards then we should not migrate that shard. There should be some logging to indicate that the shard was not migrated.

Definition of Done
  • Enable the migrate-hot-shards flag so it doesn't return an error;
  • Implement a technique for detecting a hot shard;
  • Add some test cases.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the influxd migrate entry point and inspect how the --migrate-hot-shards flag and shard WAL files are handled. Review the Engine IsIdle() API for context and run or add migration test cases. Done means hot shards are skipped by default, the flag permits migration without an error, and logging explains skipped shards.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.