Local object store behavior for Put of already existing objects

Is your feature request related to a problem? Please describe.

common.Storage interface exposes a Put() function that's supposed to be pushing objects into the store. One can Put an object and then Put it again. What happens next? Currently:

• Put succeeds, returning nil
• data is overwritten (FSTree/Peapod/bbcz)

In general we suppose that objects are immutable and we have the same data for the same object. Which means that the first point is likely correct. If that's the case then the second one doesn't matter. However if you're to change the logic of FSTree to preserve an already existing file TestRefillMetabaseCorrupted suddenly starts to fail because it relies on this overwriting semantics.

Describe the solution you'd like

We need to specify what the proper behavior is for all stores.

Describe alternatives you've considered

And it's not that easy, because:

• we can have concurrent writers pushing the same data, these prefer to see nil and don't care about overwriting, that's the most common case
• old data is supposed to be good, so overwriting is not necessary
• we can also have errors leading to different data being pushed for the same address, in this case keeping old data is also more safe
• but in this case returning nil looks bad since writer would think that he has successfully pushed something into the store, but any Get would reveal that it's not exactly true (the data is different)
• if we're to overwrite then nil makes total sense
• old data can be corrupted as well, so we may want to do a forced write (although it can be done with Delete/Put)
• checking data consistency and returning different status based on that sounds nice, but can lead to substantial overhead that is not justified for the most common case
• always returning EEXIST is an option (let the upper layer handle), but can be daunting for callers

So, what do we really want from our stores?

Start with the common.Storage Put contract and compare the existing behavior of FSTree, Peapod, and bbcz for repeated writes. Read TestRefillMetabaseCorrupted to understand its reliance on overwriting. Done means the repeated-Put behavior is agreed for all stores and the relevant tests reflect that decision.