openshift / openshift/openshift-velero-plugin

Image stream backup: Azure Workload Identity (WIF) not supported in registry env var wiring

Open
#439 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
55
Forks
45
Avg merge
3d 10h
Merged PRs (30d)
11

Description

Summary

Image stream backup on Azure Workload Identity (WIF) clusters does not work because getAzureRegistryEnvVars() always populates REGISTRY_STORAGE_AZURE_ACCOUNTKEY and SPN secret references, which prevents the openshift/docker-distribution Azure storage driver from falling through to azidentity.NewDefaultAzureCredential() — the code path that supports Workload Identity.

AWS STS and GCP WIF image stream backup already work end-to-end. Azure is the only remaining gap.

Background

OpenShift supports short-term credentials for all three cloud providers via CCO (Cloud Credential Operator):

  • AWS STSrole_arn + web_identity_token_file
  • GCP WIFexternal_account JSON with federated token source
  • Azure WIAZURE_CLIENT_ID + AZURE_TENANT_ID + AZURE_FEDERATED_TOKEN_FILE

See: https://docs.okd.io/latest/authentication/managing_cloud_provider_credentials/cco-short-term-creds.html

The OADP operator already provisions STS/WIF credentials for all three providers via pkg/credentials/stsflow/stsflow.go. The Velero object storage plugins all support short-term tokens. The openshift/docker-distribution Azure storage driver supports DefaultAzureCredential (which includes WorkloadIdentityCredential).

The only missing piece is the wiring in this repository's getAzureRegistryEnvVars() function.

How it works today (broken for WIF)

velero-plugins/imagestream/registry.go getAzureRegistryEnvVars() always sets:

  • REGISTRY_STORAGE_AZURE_ACCOUNTKEY → from secret storage_account_key
  • REGISTRY_STORAGE_AZURE_SPN_CLIENT_ID → from secret client_id_key
  • REGISTRY_STORAGE_AZURE_SPN_CLIENT_SECRET → from secret client_secret_key
  • REGISTRY_STORAGE_AZURE_SPN_TENANT_ID → from secret tenant_id_key

On WIF clusters, these secret keys don't exist (the credential secret contains AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID instead). This causes pod startup failures or incorrect auth.

How the docker-distribution Azure driver authenticates

In openshift/docker-distribution release-4.19 azure_auth.go, newAzureClient() has three auth paths:

func newAzureClient(params *Parameters) (*azureClient, error) {
    if params.AccountKey != "" {
        // Path 1: SharedKeyCredential (static storage account key)
    }

    if params.Credentials.Type == "client_secret" {
        // Path 2: ClientSecretCredential (SPN with long-lived secret)
    } else {
        // Path 3: DefaultAzureCredential — includes WorkloadIdentityCredential
        cred, err = azidentity.NewDefaultAzureCredential(nil)
    }
}

Path 3 is what we need for WIF. azidentity.NewDefaultAzureCredential picks up AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_FEDERATED_TOKEN_FILE environment variables automatically. But currently, Path 1 or Path 2 is always triggered because the plugin always sets ACCOUNTKEY and SPN env vars.

How AWS STS already works (reference pattern)

The AWS path in getAWSRegistryEnvVars() correctly handles STS:

// if credential is sts, then add sts specific env vars
if bsl.Spec.Config[enableSharedConfig] == "true" {
    awsEnvs = append(awsEnvs, corev1.EnvVar{
        Name:  RegistryStorageS3CredentialsConfigPathEnvVarKey,
        Value: getBslSecretPath(bsl),
    })
} else {
    // static access key / secret key path
}

This was enabled by the S3 driver patch openshift/docker-distribution@bb88bf20 ("allow pointing to an AWS config file as a parameter for the s3 driver").

How GCP WIF already works

The GCS driver patch openshift/docker-distribution@32b33ab6 ("Allow google oauth client to consume creds for workload identity") changed JWTConfigFromJSONCredentialsFromJSON so external_account type JSON files work.

The getGCPRegistryEnvVars() passes the BSL secret path as REGISTRY_STORAGE_GCS_KEYFILE, and CredentialsFromJSON handles both service_account and external_account types transparently.

Proposed fix

Modify getAzureRegistryEnvVars() to detect WIF mode and skip ACCOUNTKEY/SPN env vars, allowing the docker-distribution Azure driver to fall through to DefaultAzureCredential:

func getAzureRegistryEnvVars(bsl *velerov1.BackupStorageLocation, azureEnvVars []corev1.EnvVar) ([]corev1.EnvVar, error) {
    if bsl.Spec.Config == nil {
        bsl.Spec.Config = make(map[string]string)
    }

    // Base env vars needed for all auth modes
    result := []corev1.EnvVar{
        {Name: RegistryStorageEnvVarKey, Value: Azure},
        {Name: RegistryStorageAzureContainerEnvVarKey, Value: bsl.Spec.StorageType.ObjectStorage.Bucket},
        {Name: RegistryStorageAzureAccountnameEnvVarKey, Value: bsl.Spec.Config[StorageAccount]},
    }

    // Detect WIF: credential secret format has AZURE_CLIENT_ID etc.
    // but no storage_account_key or client_secret.
    // When WIF is active, do NOT set ACCOUNTKEY or SPN env vars.
    // DefaultAzureCredential in the docker-distribution azure driver
    // picks up AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE
    // from the pod environment automatically.
    if isAzureWIF(bsl) {
        // Mount the WIF env vars (AZURE_CLIENT_ID, AZURE_TENANT_ID, 
        // AZURE_FEDERATED_TOKEN_FILE) into the registry container.
        // The docker-distribution azure driver's newAzureClient() will 
        // fall through to azidentity.NewDefaultAzureCredential().
        return result, nil
    }

    // Non-WIF: existing ACCOUNTKEY + SPN logic
    // ...existing code...
}

The WIF detection could check:

  • Whether the BSL credential secret contains AZURE_FEDERATED_TOKEN_FILE
  • Whether the OADP operator created an STS-labeled secret (oadp.openshift.io/secret-type: sts-credentials)
  • A new BSL config flag (similar to AWS's enableSharedConfig)

Additionally, the WIF env vars (AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE) need to be available in the Velero pod environment. These are typically projected by the OLM/CCO infrastructure via projected service account token volumes.

Related components

Component Relevant code STS/WIF status
OADP Operator pkg/credentials/stsflow/stsflow.goCreateOrUpdateSTSAzureSecret() ✅ Creates Azure WIF secret
docker-distribution Azure driver azure_auth.gonewAzureClient() falls through to DefaultAzureCredential ✅ Supports WIF via azidentity
docker-distribution S3 driver openshift/docker-distribution@bb88bf20CredentialsConfigPath param ✅ AWS STS works
docker-distribution GCS driver openshift/docker-distribution@32b33ab6CredentialsFromJSON for external_account ✅ GCP WIF works
velero-plugin-for-microsoft-azure azure_auth.go — uses azidentity ✅ Velero BSL supports WIF
openshift-velero-plugin getAzureRegistryEnvVars() in registry.go This issue — no WIF path

Acceptance criteria

  • getAzureRegistryEnvVars() detects Azure WIF mode
  • When WIF: only set container + account name, skip ACCOUNTKEY/SPN env vars
  • WIF env vars (AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE) available in registry container
  • Image stream backup succeeds on Azure WIF clusters
  • Image stream restore succeeds on Azure WIF clusters
  • Existing non-WIF Azure auth (account key, SPN) continues to work

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in velero-plugins/imagestream/registry.go at getAzureRegistryEnvVars(), then compare the AWS and GCP registry environment-variable paths. Read pkg/credentials/stsflow/stsflow.go and the referenced Azure driver authentication flow to understand the credential formats. Done means Azure WIF image-stream backup and restore work while existing account-key and SPN authentication remains functional.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go
Domain
authentication, backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.