Remove eleveldb

riak_core has a dependency on elevledb as a backend for hashtree.

There are two uses for hashtree:

• In legacy vnode AAE in riak_kv;
• To provide AAE for riak_core_metadata.

There exists an alternative to legacy vnode AAE in Tictac AAE - which is part of the nextgenrepl solution. There is currently no alternative for AAE in riak_core_metadata.

There are two options for removing eleveldb:

• i) Replace the eleveldb integration in hashtree with leveled (similar to how leveled_so is used in Tictac AAE).
• ii) Create a new simpler hashtree and refactor riak_core_metadata_exchange. That is to say, remove the need for keys/hashes compares and do more item-by-tem comparisons. Given the limited size of riak_core_metadata, the use of 1M segment merkle trees backend by to-disk key/hash stores seems excessive.

As the eleveldb stores are destroyed during restarts, then (i) offers a simple transition. There also exists the possibility that leveled in head_only mode may perform sufficiently well to be used as a backend for vnode AAE.

As an aside, there is also a third AAE implementation in Riak in riak ensemble - https://github.com/nhs-riak/riak_ensemble/blob/nhse-develop/src/synctree.erl - that is tested for orddict/ets backends as well as leveldb backends. It is assumed that this isn't a viable alternative as it has specific requirements related to a role in a strongly consistent store.

Start by reading the hashtree integration and riak_core_metadata_exchange, then compare the two proposed removal paths: using leveled or creating a simpler hashtree. The riak_ensemble synctree implementation is mentioned as a possible reference. Done means eleveldb is no longer required while AAE support for riak_core_metadata remains functional.