facebook / facebook/rocksdb

No way to have a custom comparator with a filter policy in RocksDB 7.x

Open
#10,256 1 comment 0 reactions 1 assignee Claimed by @pdillinger View on GitHub
design discussion up-for-grabs
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

When using a custom comparator that considers keys with different byte contents equal, it's impossible to use a built-in filter policy because it'll only hash a specific representation of the key and would be unable to find it when provided a different but equivalent (from the comparator's point of view) representation (see #8482).

Prior to RocksDB 7.0 it was possible to work around that issue by creating a custom policy that delegated to the built-in filter policy with the canonical representation of key. However, after the changes in #9592 it's impossible to do that because `FilterBitsBuilder` and `FilterBitsReader` are now private.

### Expected behavior
Being able to have a custom comparator with a filter policy.

### Actual behavior
For custom comparators that consider different key representations equal there's no way to set a filter policy.

### Steps to reproduce the behavior

Adapted from the `db_test` custom comparator test:

```c++
TEST_F(DBTest, CustomComparator) {
class NumberComparator : public Comparator {
public:
const char* Name() const override { return "test.NumberComparator"; }
int Compare(const Slice& a, const Slice& b) const override {
return ToNumber(a) - ToNumber(b);
}
void FindShortestSeparator(std::string* s, const Slice& l) const override {
ToNumber(*s); // Check format
ToNumber(l); // Check format
}
void FindShortSuccessor(std::string* key) const override {
ToNumber(*key); // Check format
}

private:
static int ToNumber(const Slice& x) {
// Check that there are no extra characters.
EXPECT_TRUE(x.size() >= 2 && x[0] == '[' && x[x.size() - 1] == ']')
<< EscapeString(x);
int val;
char ignored;
EXPECT_TRUE(sscanf(x.ToString().c_str(), "[%i]%c", &val, &ignored) == 1)
<< EscapeString(x);
return val;
}
};
NumberComparator cmp;
Options new_options;
new_options.create_if_missing = true;
new_options.comparator = &cmp;
new_options.filter_policy.reset(NewBloomFilterPolicy(10));
DestroyAndReopen(new_options);

ASSERT_OK(Put(1, "[10]", "ten"));
ASSERT_OK(Put(1, "[0x14]", "twenty"));
ASSERT_OK(Flush());

ASSERT_EQ("ten", Get(1, "[10]"));
ASSERT_EQ("ten", Get(1, "[0xa]"));
ASSERT_EQ("twenty", Get(1, "[20]"));
ASSERT_EQ("twenty", Get(1, "[0x14]"));
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.