[filebeat] azure-eventhub input: support DefaultAzureCredential / workload identity federation
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
## Describe the enhancement
The `azure-eventhub` input (processor v2) currently supports three `auth_type` values: `connection_string`, `client_secret`, and `managed_identity`. The managed-identity path constructs `azidentity.NewManagedIdentityCredential` directly (`x-pack/filebeat/input/azureeventhub/managed_identity.go`), which only works on Azure-hosted infrastructure (IMDS/App Service/Arc endpoints).
Please add an auth_type backed by `azidentity.NewDefaultAzureCredential` (or explicitly `NewWorkloadIdentityCredential`), applied to both the Event Hub consumer client and the checkpoint blob storage client.
## Describe a specific use case
Consuming Entra ID / Azure activity logs from Kubernetes clusters **outside Azure** (GKE/EKS/on-prem) using [workload identity federation](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation): a federated identity credential on an App Registration trusts the cluster's OIDC issuer, and the pod's projected service account token (audience `api://AzureADTokenExchange`) is exchanged for Entra tokens. The Azure SDK supports this natively via `WorkloadIdentityCredential`, which reads `AZURE_CLIENT_ID` / `AZURE_TENANT_ID` / `AZURE_FEDERATED_TOKEN_FILE` - and `DefaultAzureCredential` includes it in its chain.
With such support, Event Hub consumption needs **no secrets at all** (no SAS connection strings, no storage account keys, no client secrets to rotate) from any Kubernetes cluster with an OIDC issuer. `client_secret` auth gets close (RBAC instead of SAS) but still requires a rotatable secret.
## Current workaround
We run a small sidecar that emulates the App Service managed-identity endpoint (the `IDENTITY_ENDPOINT`/`IDENTITY_HEADER` protocol that `ManagedIdentityCredential` probes) and mints tokens via `WorkloadIdentityCredential` from the projected token. It works, but it adds an unsupported moving part to every deployment; native support would remove it.
Happy to provide more detail or test a build.
Contributor guide
Research direction
Start in x-pack/filebeat/input/azureeventhub/managed_identity.go and trace how auth_type reaches the Event Hub consumer and checkpoint blob storage clients. Compare the existing managed-identity construction with the Azure SDK's DefaultAzureCredential or WorkloadIdentityCredential; done means the new authentication mode works for both clients without secrets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go
- Domain
- authentication, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100