apache / apache/kvrocks

should we start the compaction_checker_range thread if it's not configed.

Open
#2,305 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.

### Motivation

When running Server.start(), kvrocks will launch a thread for compaction check. Some considerations are as follows.

- If there is no checker configured, maybe it is no need to start the thread to check frequently. Only if it is enabled, then create the thread.

- The checker for pubsubfiles is independent of the configuration of the normal checker (the code is like this). If there is no configured checker, kvrocks also should do compression for pubsub. (not sure I understand that correctly.)

Corresponding code block is in server.cc, Server().Start().
```
compaction_checker_thread_ = GET_OR_RET(util::CreateThread("compact-check", [this] {
uint64_t counter = 0;
int64_t last_compact_date = 0;
CompactionChecker compaction_checker{this->storage};

while (!stop_) {
// Sleep first
std::this_thread::sleep_for(std::chrono::milliseconds(100));

// To guarantee accessing DB safely
auto guard = storage->ReadLockGuard();
if (storage->IsClosing()) continue;

if (!is_loading_ && ++counter % 600 == 0 // check every minute
&& config_->compaction_checker_range.Enabled()) {
auto now_hours = util::GetTimeStamp();
if (now_hours >= config_->compaction_checker_range.start &&
now_hours <= config_->compaction_checker_range.stop) {
std::vector cf_names = {engine::kMetadataColumnFamilyName, engine::kSubkeyColumnFamilyName,
engine::kZSetScoreColumnFamilyName, engine::kStreamColumnFamilyName};
for (const auto &cf_name : cf_names) {
compaction_checker.PickCompactionFiles(cf_name);
}
}
// compact once per day
if (now_hours != 0 && last_compact_date != now_hours / 24) {
last_compact_date = now_hours / 24;
compaction_checker.CompactPropagateAndPubSubFiles();
}
}
}
}));
```

### Solution

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in server.cc at Server::Start and read the compaction_checker_thread_ setup, then trace CompactionChecker::CompactPropagateAndPubSubFiles and the compaction_checker_range configuration. Clarify the independent pubsubfiles behavior before changing thread creation; done means disabled checking does not start unnecessary work while required pubsub compaction still occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.