influxdata / influxdata/influxdb
Ensure influxd migrate tool doesn't migrate hot shards by default
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
The desirable behaviour for the influxd migrate tool is that it won't import hot shards. Importing hot shards can leave data behind inside of the in-memory cache and WAL files.
The tool has a flag that let's the operator specify they are OK with migrating hot shards (--migrate-hot-shards), but currently the tool does not actually check if a shard is hot.
We have APIs to check if a shard is hot, e.g., IsIdle() in the Engine code, but we can't use that when influxd is not running. Instead I propose that we simply check for none-truncated WAL file segments in the shard. If we detect any, and the user has not specified --migrate-hot-shards then we should not migrate that shard. There should be some logging to indicate that the shard was not migrated.
Definition of Done
- Enable the
migrate-hot-shardsflag so it doesn't return an error; - Implement a technique for detecting a hot shard;
- Add some test cases.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the influxd migrate entry point and inspect how the --migrate-hot-shards flag and shard WAL files are handled. Review the Engine IsIdle() API for context and run or add migration test cases. Done means hot shards are skipped by default, the flag permits migration without an error, and logging explains skipped shards.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100