elastic / elastic/beats

[filebeat] azure-eventhub input: support DefaultAzureCredential / workload identity federation

Open
#51,992 2 comments 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

## Describe the enhancement

The `azure-eventhub` input (processor v2) currently supports three `auth_type` values: `connection_string`, `client_secret`, and `managed_identity`. The managed-identity path constructs `azidentity.NewManagedIdentityCredential` directly (`x-pack/filebeat/input/azureeventhub/managed_identity.go`), which only works on Azure-hosted infrastructure (IMDS/App Service/Arc endpoints).

Please add an auth_type backed by `azidentity.NewDefaultAzureCredential` (or explicitly `NewWorkloadIdentityCredential`), applied to both the Event Hub consumer client and the checkpoint blob storage client.

## Describe a specific use case

Consuming Entra ID / Azure activity logs from Kubernetes clusters **outside Azure** (GKE/EKS/on-prem) using [workload identity federation](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation): a federated identity credential on an App Registration trusts the cluster's OIDC issuer, and the pod's projected service account token (audience `api://AzureADTokenExchange`) is exchanged for Entra tokens. The Azure SDK supports this natively via `WorkloadIdentityCredential`, which reads `AZURE_CLIENT_ID` / `AZURE_TENANT_ID` / `AZURE_FEDERATED_TOKEN_FILE` - and `DefaultAzureCredential` includes it in its chain.

With such support, Event Hub consumption needs **no secrets at all** (no SAS connection strings, no storage account keys, no client secrets to rotate) from any Kubernetes cluster with an OIDC issuer. `client_secret` auth gets close (RBAC instead of SAS) but still requires a rotatable secret.

## Current workaround

We run a small sidecar that emulates the App Service managed-identity endpoint (the `IDENTITY_ENDPOINT`/`IDENTITY_HEADER` protocol that `ManagedIdentityCredential` probes) and mints tokens via `WorkloadIdentityCredential` from the projected token. It works, but it adds an unsupported moving part to every deployment; native support would remove it.

Happy to provide more detail or test a build.

Contributor guide

Open the contributing guide

Research direction

Start in x-pack/filebeat/input/azureeventhub/managed_identity.go and trace how auth_type reaches the Event Hub consumer and checkpoint blob storage clients. Compare the existing managed-identity construction with the Azure SDK's DefaultAzureCredential or WorkloadIdentityCredential; done means the new authentication mode works for both clients without secrets.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go
Domain
authentication, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.