Support ability to export to multiple tables and guarantee consistency of exports.

This stems from an offline discussion with Keith. I have a couple of tables which use sharded document partitioned indexing to allow robust and scalable query of documents. The document table contains two different types of key/values- a field index, or an inverted index, to find a document id given the indexed fields upon which it is composed, and the document itself, which is serialized into the value. The global index table contains 2 different types of key/values as well- one for which shards contain which keys, and another for which shards contain which values. The key/values in the global index table both have cardinality information so that queries can be ordered to be the most effective.

The fact that the store is sharded makes it inefficient to do conditional mutations because it will lock entire rows. Since the Fluo table uses conditional mutations, we have been discussing using Fluo as a separate table for managing transactions which get exported to other tables rather than being queried directly.

Discussed was a possible design whereby 3 observers can be chained together- one to initiate the update of the document table, one to initial the update of the global indexes, and a third to act as a commit for the update to the field indexes and to switch the newly written document to "on", thereby using some flag or state that the partitioned table can use to determine that old indexes are no longer valid (it was discussed possibly using sequence numbers that align with the transaction number). An iterator may be able to determine itself which state should be used here based on said state. If an observer were to fail, the export could be retried in the case where the transaction itself has already been committed. This would work so long as the mutations being placed in all the tables are idempotent. One potential problem to this is that cardinalities are being kept in the global index stores using aggregators and retries could make these cardinalities no longer align with the actual counts of the keys/values within a set of shards. We would need to test this but it is my belief that this would actually not happen often enough to make a drastic difference in the query times.

Other possible designs discussed included using only conditional mutations and implementing a daemon that would run local to a tablet server, intercepting all conditional mutation requests for a single tablet and batching them thereby drastically reducing the number of locks needed for that tablet. This would allow for atomic updates at the document level but still would not guarantee a high level of atomicity across the mutations which need to be placed in multiple tables.

No files or tests are named. Start by reviewing the existing Fluo observer, transaction, conditional-mutation, and export mechanisms, then compare the proposed chained-observer and tablet-server approaches. Done should mean a decided design and an implementation plan that guarantees consistent, retry-safe exports across the document and global-index tables.

Written by the indexing model from the issue text.