Race conditions with key-value stores

In ocis we use key-value stores to store metadata for some services like notifications for example.

All these stores use a simple approach to modify data: Load it - Modify it - Store it. In the single binary this is not problem at all. But when running multiple instances of the same service, this could lead to race conditions: Service A loads - Service B writes - Service A writes.

We need a way to avoid that.

One way would be to use [optimistic locking](https://docs.nats.io/nats-concepts/jetstream/key-value-store/kv_walkthrough#update-with-cas-aka-optimistic-locking) . We could have a retry mechanism to avoid unnecessary errors.

Another way would be to use nats queues to build some sort of Mutex. The result would be that Service A can `Lock` the key, so that others can't read from it. Service B then needs to wait until Service A releases the lock (or timeout).

Maybe there are other options.

Start by mapping how the key-value stores are used to save metadata in services such as notifications, then compare the optimistic-locking and NATS-queue approaches described in the issue. Done means selecting and implementing a concurrency strategy that prevents lost updates across multiple service instances, including a retry or timeout behavior where appropriate.

Written by the indexing model from the issue text.