Current PMGD Handler Process Query Flow May Lead to Metadata/Data Inconsistencies

**Context and Background:**

Once you’ve entered the process query method in the PMGD based handler, when a VDMS call comes from a client, there are effectively 3 phases.

1. Constructing the query, and saving any uploaded blobs. This is the construct_protobuf call that we refer to in the RSCommands map. It is done iteratively over each VDMS command.
2. Issuing the PMGD query, using the nodes that were created via the calls to the different RSCommands construct_protobuf method. Important note, to the best of my knowledge, PMGD is only called once during the entire flow. If this phases succeeds, we dont touch PMGD for the remainder of the VDMS command(s) we're dealing with.
3. Constructing the responses. In this phase we also do any operations (e.g. resize) on data affiliated with things like Find commands before sending it back to the calling client.

**Now lets talk about failure handling.**

- If something fails during the first phase, we use the list of uploaded files (images_log, videos_log) the error handler goes through and deletes them. PMGD has yet to be called so there is no concern over metadata being written.
- If the PMGD transaction fails (second phase), there will be no PMGD data written, as PMGD itself has a concept of a transaction and everything will be rolled back. The error from PMGD will trigger a call to the VDMS error handler, deleting any data.
- However, if there is a failure in this 3rd phase, it looks like it possible to get into an inconsistent state.

- Consider a simple example of an AddImage call followed by a FindImage call that returns a blob after some operations done on it.
-- The add Image data will be written in phase 1
-- The PMGD data will be written in phase 2.
-- The Pointer to the image from the FindImage call will be obtained in phase 2.
-- In phase 3, we follow the pointer from the find image call (a file path), read the data in, and do any operations before sending the results back to the calling client

- My concern is that failures in phase 3 may cause consistency problems. Consider the example above.
-- Phase 3 fails, and error/cleanup handler is called. **_*however* this cleanup handler only removes data, it does not rollback PMGD, as the PMGD transaction at this point is “done”._** In the example above, this means that a failure in the operations associated with FindImage would lead to the previous image that was saved to disk/S3 being removed. But the PMGD information for that new image is *not* updated.
--- In Short, we wind up in a situation where the metadata store (PMGD) thinks we have saved data for a particular image that no longer exists.

What needs to happen is that a final commit of the transactions needs to occur after this last phase if we are to maintain something that looks like transactional semantics for VDMS as a whole. Correspondingly the error handler which removes written data should also be responsible for ensuring a rollback/aborted transaction.

However, naively leaving the transaction open until other operations have completed can cause an unacceptable performance drop due to lock contention within the graph store.