Ability to do optimistic locking for resource metadata

Issue

When updating metadata on CF API resources, its not possible to do optimistic locking. This results in race conditions when updating metadata from multiple clients concurrently.

Context

We would like to use metadata for storing some information on CF API resources, such as routes. The information will be updated by multiple microservice instances which do not have the ability to do pessimistic locking by themselves. When reading and updating the metadata, we run into classic race conditions where one service overwrites the data of another. The principle is outlined nicely in this article: https://sookocheff.com/post/api/optimistic-locking-in-a-rest-api/

Our current goal is to implement usage tracking on resources using the metadata, such that we can ensure a resource is only deleted when all usages are removed. In this scenario its crucial to avoid race conditions to ensure that no usage is added once the decision to delete a resource is made.

It should be possible for a client to ensure that the metadata has not changed since it has fetched it from the API when submitting an update. Ideally, this would extend to more operations than just updating the metadata, such as deleting (e.g. only delete a resource when the metadata is in the expected state).

Possible Fix

One possible solution is to add a mechanism similar to etags to the metadata, that changes on every metadata update. A client can then supply an If-Match header containing that value and the CloudController would only update the metadata when the etag value matches, otherwise it would return an error. This would optimistically lock the whole metadata at once.

A more fine-grained but also more complex solution would be to pass an expected key-value pair in this If-Match header (possibly even multiple pairs in multiple headers) that need to match for the request to succeed. This way its possible to lock only a part of the metadata, allowing updates to unrelated metadata unaffected.

Start by tracing the Cloud Controller API entry points for resource metadata updates and deletes, then review the proposed If-Match and ETag behavior in the issue and linked article. The issue does not name files or tests; work is not ready to start until the locking scope, failure response, and supported operations are decided.