Object de-duplication for storage and replication efficiency

In order to facilitate efficient modes of replicating objects between EDV instances a core optimisation would be enabling the capability for an EDV to appropriately identify duplicate objects. With solutions that deal with un-encrypted data this is a relatively straight forward exercise using techniques like hash based content addressing and is at the core of systems like git. However when the data being replicated is in an encrypted form, things get more complicated.

**Why?** Most encryption schemes on purpose do not produce deterministic ciphers for the same payload for security reasons. In layman's terms this means the exact same payload encrypted twice will yield two difference ciphers. Meaning w.r.t EDV's, in an instance where the same object is encrypted on two seperate occasions and inserted into two seperate EDV's that are connected via replication, the encrypted payload will be persisted on disk twice in both vaults. The vaults will also unnecessarily send a copy of the same encrypted object to one an other. The below diagram hopefully captures this.

**Potential Solutions**

1. Deterministic Encryption Schemes - Some approaches use [Synthetic Initialization Vectors](https://tools.ietf.org/html/rfc5297) that derive the IV from HMACing the content to be encrypted hence creating a stable resulting cipher for the same content. See https://github.com/AGWA/git-crypt#security as an example.

2. Using indexes - @OR13 raised this as a possible solution, @OR13 could you elaborate?

3. Not caring - Essentially opting to live with the cases where duplication will occur and accepting the trade-offs that it creates.

I'm sure there are other possible solutions for this problem and @dlongley when we spoke briefly about [Synthetic Initialization Vectors](https://tools.ietf.org/html/rfc5297) you raised some good questions around the security of content encryption ciphers when used in this mode.

Another observation w.r.t a deterministic encryption scheme, is that if the content encryption key is rotated (i.e the object is re-encrypted) then the cipher will change anyway.