apache / apache/kvrocks

Large .sst and .log files (unsure about compression feature)

Open
#2,365 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.

### Motivation

I was playing with kvrocks (version 2.8.0) today and my goal was to activate compression and reduce logging.
I am not sure if I have enabled configurations correctly but here is what I have noticed.

My goal was to use kvrocks and storing large volumes of KVs while ensuring that that data (KVs) remain compressed and uses least amount of disk.

Here is my configuration:

```
dir /kvrocks/data

# General
port 6376
bind 0.0.0.0

rocksdb.compression zstd

backup-dir /kvrocks/backup
#log-dir /kvrocks
log-level error
log-dir stdout
log-retention-days 0
pidfile /kvrocks/kvrocks.pid
```

I tried also the option lz4 on rocksdb.compression but nothing, the .sst file does not look compressed and the .log files are quite large as well.
cd db
/kvrocks/data/db # du -hs *
4.0K 000019.sst
624.0K 000605.sst
83.9M 002080.sst
70.4M 002082.log
102.6M 002083.sst
16.0G archive

cd archive
ls -alsrht

64544 -rw-r--r-- 1 root root 63.0M Jun 14 20:59 002162.log
64448 -rw-r--r-- 1 root root 62.9M Jun 14 20:59 002169.log
64696 -rw-r--r-- 1 root root 63.2M Jun 14 20:59 002165.log

127.0.0.1:6376> keys '*' pattern
16521) ....

it has 16521 records.

16GB is just too much.

### Solution

I am not sure on how it should work and be implemented, but what I expected as user is to see .sst files compressed and be able to disable logs if desired.

instead of seeing:

/kvrocks/data # du -hs *
22.5G db

it should have been something like: 25MB

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported configuration with rocksdb.compression, log-dir, log-level, and log-retention-days, then inspect the data/db and archive directories. Determine whether SST and WAL log growth matches the configured compression and logging behavior; done means the expected disk usage and log controls are documented or corrected.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.