Azure / Azure/azure-sdk-tools

Capture the need to rotate relevant openapi-alps secrets in the secrets management tool, including `pipeline-storage-container-sat`

Open
#7,248 0 comments 1 reaction 1 assignee Claimed by @konrad-jamrozik View on GitHub
Central-EngSys Spec PR Tools
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
144

Description

openapi-alps has many secrets expiring at random times, which so far caused at least two incidents:

- https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/1019/Incident-post-mortem-11-06-2023-SAS-URL-expired-blocking-all-spec-PRs
- https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/997/Incident-post-mortem-8-11-2023-pipeline-bot-cert-expired-blocking-all-specs-PRs

and also almost third one, but we caught it literally 2 days in advance [1]

Most notably, we know we will have to rotate the secret `pipeline-storage-container-sat` by 11/6/2025, otherwise we will have another incident. Details of that are given in:

- https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/1019/Incident-post-mortem-11-06-2023-SAS-URL-expired-blocking-all-spec-PRs

We want to ensure the information about all openapi-alps secrets, including when and how to rotate, is captured in the [secrets management tool](https://dev.azure.com/azure-sdk/internal/_git/azure-sdk-build-tools?path=/tools/secret-management) written by Patrick. There is also a bit more [possibly relevant, outdated docs](https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/775/Secrets-Rotation) that may be helpful here.

Note some of the secrets to capture there are listed on the [pipeline-bot secrets page](https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/4e7b30e5-96b6-4d26-ae34-bd0b75fdafb4/resourceGroups/rg-container-apps-test/providers/Microsoft.App/containerapps/pipeline-bot/secretsList). The following work item also mentions more:
- https://github.com/Azure/azure-sdk-tools/issues/7249

We should systematically discover, review and capture all openapi-alps secrets sources.

[1] On 11/2/2023 We observed in TEST environment the following:

> pipeline-eventhub-cs: "Could not fetch access token for Azure. Status code: invalid_client, status message: 7000222 - [2023-11-02 23:23:34Z]: AADSTS7000222: The provided client secret keys for app '***' are expired.

and saw the PROD secret will expire in two days. Wes created new principals to prevent the incident.

@weshaggard @mikeharder @hallipr FYI

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.