elastic / elastic/integrations

[Kafka Consumer (OpenTelemetry)]: Add azure_auth extension support

Open
#20,480 1 comment 0 reactions 0 assignees View on GitHub
needs:triage
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Integration Name

Kafka Consumer (OpenTelemetry) [packages/kafka_input_otel]

### Dataset Name

kafkareceiver

### Integration Version

0.1.1

### Agent Version

9.4.4

### OS Version and Architecture

N/A

### User Goal

Starting with Elastic-Agent 9.3.6 and 9.4.3 (https://github.com/elastic/elastic-agent/pull/14534), Elastic-Agent includes the Azure Auth extension that can be used to authenticate via Azure Entra ID.

However, this extension is currently not usable in Fleet because it should be used by the relevant integrations.

We would like to achieve the following configuration:

```
extensions:
azure_auth:
service_principal:
client_id: "${env:CLIENT_ID}"
tenant_id: "tenant-id"
client_secret: "${env:CLIENT_SECRET}"
scopes:
- "api://confluent-cloud/.default"

receivers:
kafka:
brokers:
- "${env:GATEWAY_KAFKA_BROKER}"
protocol_version: "2.1.0"
logs:
topics:
- "${env:KAFKA_LOGS_TOPIC}"
metrics:
topics:
- "${env:KAFKA_METRICS_TOPIC}"
traces:
topics:
- "${env:KAFKA_TRACES_TOPIC}"
group_id: "${env:KAFKA_CONSUMER_GROUP}"
tls:
insecure: false
insecure_skip_verify: false
auth:
sasl:
mechanism: OAUTHBEARER
version: 0
oauthbearer_token_source: azure_auth

...

service:
extensions:
- azure_auth
pipelines:
...
```

### Existing Features

The current feature does not have support for azure_auth extension.

### What did you see?

N/A

### Anything else?

N/A

Contributor guide

Open the contributing guide

Research direction

Start in packages/kafka_input_otel and trace how the Kafka consumer currently handles authentication configuration. Compare that path with the requested azure_auth extension and service wiring; done means the shown configuration is accepted and the consumer can use Azure Entra ID authentication.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kafka
Domain
authentication, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.