org.rocksdb.RocksDBException: Compaction sees out-of-order keys
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev
We have observed that our applications failed with Rocksdb `Compaction sees out-of-order keys` issue after we bumped the Java Rocksdb client version from `6.6.4` to `7.0.3`. The issue is similar as this existing reported issue [8248](https://github.com/facebook/rocksdb/issues/8248)
We noticed that since the version `6.14` Rocksdb added [a new feature](https://github.com/facebook/rocksdb/commit/750817555867a43f0e7b73dffa44756a9136c808) `check_flush_compaction_key_order`, and I'm suspecting this feature caused our applications failed because the new feature is enabled by default.
I wanted to see if I can use Java Rocksdb client to disable this new feature, but seems there is no method available like `options.setCheckFlushCompactionKeyOrder(false)` for testing.
I would like to check with Rocksdb community a few questions:
- What kind of scenarios that Rocksdb can produce such out-of-order keys?
- Can this feature `check_flush_compaction_key_order ` cause the error `Compaction sees out-of-order keys`?
- Do we support a way to disable this `check_flush_compaction_key_order` feature in Java Rocksdb client?
### Expected behavior
The application should not be failed with this issue after Rocksdb version bump
### Actual behavior
The application failed with below exception after bumped up Rocksdb version from `6.6.4` to `7.0.3`
```
Caused by: org.rocksdb.RocksDBException: Compaction sees out-of-order keys.
at org.rocksdb.RocksDB.write0(Native Method) ~[org.rocksdb-rocksdbjni-7.0.3.jar:?]
at org.rocksdb.RocksDB.write(RocksDB.java:1706) ~[org.rocksdb-rocksdbjni-7.0.3.jar:?]
at org.apache.samza.storage.kv.RocksDbKeyValueStore.$anonfun$putAll$1(RocksDbKeyValueStore.scala:206) ~[com.linkedin.samza-li-samza-kv-rocksdb_2.12-323.1085.2.13.jar:?]
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) ~[org.scala-lang-scala-library-2.12.11.jar:?]
at org.apache.samza.storage.kv.RocksDbKeyValueStore.org$apache$samza$storage$kv$RocksDbKeyValueStore$$ifOpen(RocksDbKeyValueStore.scala:290) ~[com.linkedin.samza-li-samza-kv-rocksdb_2.12-323.1085.2.13.jar:?]
at org.apache.samza.storage.kv.RocksDbKeyValueStore.putAll(RocksDbKeyValueStore.scala:187) ~[com.linkedin.samza-li-samza-kv-rocksdb_2.12-323.1085.2.13.jar:?]
```
### Steps to reproduce the behavior
Contributor guide
Research direction
Start with the referenced issue 8248 and commit 750817555867a43f0e7b73dffa44756a9136c808, then compare the Java RocksDB options API between versions 6.6.4 and 7.0.3. Trace the reported failure through RocksDB.write and determine whether check_flush_compaction_key_order explains it or can be disabled from Java; done means a confirmed cause and supported resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100