facebook / facebook/rocksdb

org.rocksdb.RocksDBException: Compaction sees out-of-order keys

Open
#12,119 5 comments 0 reactions 0 assignees View on GitHub
java java-api
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

We have observed that our applications failed with Rocksdb `Compaction sees out-of-order keys` issue after we bumped the Java Rocksdb client version from `6.6.4` to `7.0.3`. The issue is similar as this existing reported issue [8248](https://github.com/facebook/rocksdb/issues/8248)

We noticed that since the version `6.14` Rocksdb added [a new feature](https://github.com/facebook/rocksdb/commit/750817555867a43f0e7b73dffa44756a9136c808) `check_flush_compaction_key_order`, and I'm suspecting this feature caused our applications failed because the new feature is enabled by default.

I wanted to see if I can use Java Rocksdb client to disable this new feature, but seems there is no method available like `options.setCheckFlushCompactionKeyOrder(false)` for testing.

I would like to check with Rocksdb community a few questions:

- What kind of scenarios that Rocksdb can produce such out-of-order keys?
- Can this feature `check_flush_compaction_key_order ` cause the error `Compaction sees out-of-order keys`?
- Do we support a way to disable this `check_flush_compaction_key_order` feature in Java Rocksdb client?

### Expected behavior
The application should not be failed with this issue after Rocksdb version bump

### Actual behavior
The application failed with below exception after bumped up Rocksdb version from `6.6.4` to `7.0.3`

```
Caused by: org.rocksdb.RocksDBException: Compaction sees out-of-order keys.
at org.rocksdb.RocksDB.write0(Native Method) ~[org.rocksdb-rocksdbjni-7.0.3.jar:?]
at org.rocksdb.RocksDB.write(RocksDB.java:1706) ~[org.rocksdb-rocksdbjni-7.0.3.jar:?]
at org.apache.samza.storage.kv.RocksDbKeyValueStore.$anonfun$putAll$1(RocksDbKeyValueStore.scala:206) ~[com.linkedin.samza-li-samza-kv-rocksdb_2.12-323.1085.2.13.jar:?]
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) ~[org.scala-lang-scala-library-2.12.11.jar:?]
at org.apache.samza.storage.kv.RocksDbKeyValueStore.org$apache$samza$storage$kv$RocksDbKeyValueStore$$ifOpen(RocksDbKeyValueStore.scala:290) ~[com.linkedin.samza-li-samza-kv-rocksdb_2.12-323.1085.2.13.jar:?]
at org.apache.samza.storage.kv.RocksDbKeyValueStore.putAll(RocksDbKeyValueStore.scala:187) ~[com.linkedin.samza-li-samza-kv-rocksdb_2.12-323.1085.2.13.jar:?]
```

### Steps to reproduce the behavior

Contributor guide

Open the contributing guide

Research direction

Start with the referenced issue 8248 and commit 750817555867a43f0e7b73dffa44756a9136c808, then compare the Java RocksDB options API between versions 6.6.4 and 7.0.3. Trace the reported failure through RocksDB.write and determine whether check_flush_compaction_key_order explains it or can be disabled from Java; done means a confirmed cause and supported resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.