google / google/leveldb

logfile growing without bound if we keep writing empty WriteBatch (1.20)

Open
#911 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
39.4k
Forks
8.2k
PR merge metrics
No merged PRs in 30d

Description

Hi all,

I am using leveldb 1.20.0, and I found that if we keep writing empty WriteBatch. the logfile size will increase without bound.

```
#include
#include
#include
#include
#include

using std::cout;
using std::endl;
static void print_usage(const char *pname) {
cout << "Usage: " << pname << " " << endl;
}

int main(int argc, const char **argv) {
if (argc != 3) {
print_usage(argv[0]);
return -1;
}

const std::string &dbpath {argv[1]};
std::istringstream iss {argv[2]};
uint64_t cnt = 0;
if (!(iss >> cnt)) {
print_usage(argv[0]);
return -1;
}

leveldb::Options options;
options.create_if_missing = true;
options.write_buffer_size = 65536; // 64KB
leveldb::DB *db_handle {nullptr};
leveldb::Status status = leveldb::DB::Open(options, dbpath, &db_handle);

if (!status.ok()) {
cout << "Open db " << dbpath << " failed : " << status.ToString() << endl;
return -1;
}

leveldb::WriteOptions write_options;

cout << "Start to put " << cnt << " Empty KV" << endl;
for (size_t i = 0; i < cnt; i++) {
leveldb::WriteBatch wb;
leveldb::Status w_status = db_handle->Write(write_options, &wb);
if (!w_status.ok()) {
cout << "Write db failed : " << w_status.ToString() << endl;
return -1;
}
}
return 0;
}
```
```
[root@7d2b7faa12eb ~]# ./test_empty_wb_write ./empty_wb_1000 1000
Start to put 1000 Empty KV
[root@7d2b7faa12eb ~]# ./test_empty_wb_write ./empty_wb_1000000 1000000
Start to put 1000000 Empty KV
[root@7d2b7faa12eb ~]# ./test_empty_wb_write ./empty_wb_1000000000 1000000000
Start to put 1000000000 Empty KV
[root@7d2b7faa12eb ~]# ls -alh empty_wb_1000
total 40K
drwxr-xr-x 2 root root 4.0K Jun 3 05:34 .
dr-xr-x--- 6 root root 4.0K Jun 3 05:43 ..
-rw-r--r-- 1 root root 19K Jun 3 05:34 000003.log
-rw-r--r-- 1 root root 16 Jun 3 05:34 CURRENT
-rw-r--r-- 1 root root 0 Jun 3 05:34 LOCK
-rw-r--r-- 1 root root 57 Jun 3 05:34 LOG
-rw-r--r-- 1 root root 50 Jun 3 05:34 MANIFEST-000002
[root@7d2b7faa12eb ~]# ls -alh empty_wb_1000000
total 19M
drwxr-xr-x 2 root root 4.0K Jun 3 05:34 .
dr-xr-x--- 6 root root 4.0K Jun 3 05:43 ..
-rw-r--r-- 1 root root 19M Jun 3 05:34 000003.log
-rw-r--r-- 1 root root 16 Jun 3 05:34 CURRENT
-rw-r--r-- 1 root root 0 Jun 3 05:34 LOCK
-rw-r--r-- 1 root root 57 Jun 3 05:34 LOG
-rw-r--r-- 1 root root 50 Jun 3 05:34 MANIFEST-000002
[root@7d2b7faa12eb ~]# ls -alh empty_wb_100000000
total 1.8G
drwxr-xr-x 2 root root 4.0K Jun 3 05:37 .
dr-xr-x--- 6 root root 4.0K Jun 3 05:43 ..
-rw-r--r-- 1 root root 1.8G Jun 3 05:42 000003.log
-rw-r--r-- 1 root root 16 Jun 3 05:37 CURRENT
-rw-r--r-- 1 root root 0 Jun 3 05:37 LOCK
-rw-r--r-- 1 root root 57 Jun 3 05:37 LOG
-rw-r--r-- 1 root root 50 Jun 3 05:37 MANIFEST-000002

```

I think is caused by memtable size not triggering the minor compaction , but keep appending log to logfile.

Is this a known and expected behavior, or is it already fixed in the later leveldb version?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.