Proposal: support idempotency token for create silence api

Problem

We can not retry on Post silence api from client side because every put silence will generate an UUID. Ideally we should allow an idempotency token in create silence api so that an retry with the same idempotency token can be ignored“

An example of how cortex calling alert manager POST silence API:
If the POST silence call to the first alert manager that is responsible for tenant A, cortex won’t retry to next instance that responsible for tenant A as the lack of idempotency token support discussed here : https://github.com/cortexproject/cortex/pull/4422

Proposed solution

API

Introduce an idempotency token

We will accept an optional parameter for idempotency token, the behavior will remain unchanged if no idempotency token is provided

Pros:

Broadly adopted solution

Cons:

Require more complicated implementation as discussed in the Supporting of idempotency token section below

Allow client to input an UUID for the silence to be created

We will accept an optional parameter for UUID, if the UUID doesn’t exist before, the silence will be created, otherwise will be an update operation

Pros:

Easier to implement

Cons:

This is make create to become current update silence implementation, kind of confuse to the client

Supporting of idempotency token

Support without storing the idempotency token

Instead generating a unique uuid for every new request with a idempotency token here: https://github.com/prometheus/alertmanager/blob/f59460bfd4bf883ca66f4391e7094c0c1794d158/silence/silence.go#L579
We could generate an UUID based on the silence payload+idempotency token so that we make sure that same silence ID will be generated for the same idempotency+payload request.

With idempotency token in the request

1. Client request to create a silence with an idempotency token
2. A uuid(payload+idempotency) is returned => assume the uuid = 1ab
3. Another request comes in
   1. Create a silence with the same idempotency token and payload, the same UUID will be returned as 1ab
   2. Create a silence with the same idempotency token but different payload, different UUID will be returned as 1bc
   3. Create a silence with the different idempotency token but same payload, different UUID will be returned as 1cd

Without idempotency token in the request(remain unchanged behavior as it is right now)

1. Client request to create a silence without an idempotency token
2. A uuid is returned => assume the uuid = 2ab
3. Another request comes in
   1. Create a silence with the same payload, the UUID will be returned as 2bc
   2. Create a silence with the different payload, the UUID will be returned as 2de

Pros

• No extra dependency is needed
• Easier implementation

Cons

• In the case of idempotency token was send in before with different payload, we will return a different UUID and let the create go through instead of conflict exception

Support with storing the idempotency token

We will store the request payload and idempotency token to an external key value storage such as dynamodb , more storage option can be added

With idempotency token in the request

1. Client request to create a silence with an idempotency token
2. A query to the external storage to see if the idempotency token record exist
   1. If exist and payload match, An uuid(payload+idempotency) will be returned and silence will be create => assume the uuid = 3ab
   2. If exist and payload doesn’t match, a conflict exception will be returned
   3. If not exist, an record with the idempotency token will be created and uuid(payload+idempotency) will be returned => assume the uuid = 3ab
3. Another request comes in
   1. Create a silence with the same idempotency token and payload, the same UUID will be returned as 3ab
   2. Create a silence with the same idempotency token but different payload, a conflict exception will be returned
   3. Create a silence with the different idempotency token but same payload, different UUID will be returned as 3cd

Pros

• We will do proper validation regarding to existing idempotency token but with different payload

Cons

• Need extra dependency

Start with the create-silence API and the UUID-generation logic in silence/silence.go referenced by the issue. Review the proposed token-generation and external-storage alternatives, including behavior for repeated tokens and changed payloads. Done requires an agreed API and implementation approach, with validation of the selected behavior.