apache / apache/kvrocks

Delete expired big keys during first query

Open
#876 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues.

### Motivation

We encountered a slow log problem when accessing a rather small key. This key is in cold data and kvrocks took 0.82s to get it every time.
```
10.x.x.x:6379> strlen "scene:xxx:-1"
(integer) 0
(0.82s)
```
Here is the perflog. We can see it took most time in reading block phase, which processed 73752672 bytes. Every time we accessed the key, same thing happened, until a compaction.
```
10.x.x.x:6379> perflog get *
1) 1) (integer) 1
2) (integer) 1662454072
3) "strlen"
4) (integer) 633163
5) "user_key_comparison_count = 76, block_read_count = 1, block_read_byte = 73752672, block_read_time = 59590859, block_checksum_time = 9643232, block_decompress_time = 350221281, get_read_bytes = 247610038, get_snapshot_time = 883, get_from_memtable_time = 4306, get_from_memtable_count = 1, get_post_process_time = 1681, get_from_output_files_time = 426225825, new_table_block_iter_nanos = 426163267, block_seek_nanos = 36515, bloom_sst_hit_count = 1, bloom_sst_miss_count = 2, "
6) "thread_pool_id = 4, bytes_read = 73752672, read_nanos = 59588262, "
```
After discussing with developers, we found this is caused by a expired big key. In every querying, kvrocks accessed the big key to retrive its ttl, which took a lot of IO time.

### Solution

Maybe we can solve this problem by deleting these kind of expired big keys during the first querying, which should avoid accessing bigkeys every time. Or there may be a better solution. Hope you guys can solve this, thanks.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the strlen command and perflog evidence in this report, then trace how expired big keys are handled during their first query. Determine whether the first access can remove the expired key and verify that repeated accesses no longer incur the reported block-reading cost.

Written by the indexing model from the issue text.

Assessment

Tech stack
redis
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.