fluent / fluent/fluent-bit

out_opentelemetry: add cloud metadata endpoint token authentication

Open
#11,675 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

The OpenTelemetry output plugin supports OAuth2 client-credentials authentication but has no built-in way to authenticate using cloud instance metadata tokens - the standard mechanism on platforms where a VM has a linked service account.

### Use case

VMs with linked service accounts need to send OTLP data to cloud logging/monitoring backends that require IAM bearer tokens. Today the only options are a sidecar proxy (OTel Collector with an auth extension) or a custom Fluent Bit build. Both add operational complexity that a native plugin option would eliminate.

The `out_stackdriver` plugin already solves this for GCP via `gce_metadata.c`, but the OTel output plugin - which is becoming the recommended path for logs, metrics, and traces - has no equivalent.

### Proposed approach

Add three config options to `out_opentelemetry`:

| Option | Description | Default |
|--------|-------------|---------|
| `metadata_token_url` | HTTP URL of the metadata token endpoint. Presence enables the feature. | (none) |
| `metadata_token_header` | Extra header for the GET request (e.g. `Metadata-Flavor: Google`). | (none) |
| `metadata_token_refresh` | Token refresh interval in seconds (minimum 61). | `3600` |

The plugin fetches a Bearer token via HTTP GET from the configured URL, parses the standard `{ "access_token": "...", "expires_in": N }` JSON response, and populates the existing `flb_oauth2` context. Token caching, periodic refresh, and 401 retry reuse the existing oauth2 code paths - no new token injection mechanism is introduced.

The design follows the pattern established by `out_stackdriver` (`gce_metadata.c`) but is cloud-neutral: no vendor-specific naming, no compile flags, no new dependencies. It works with any metadata endpoint that returns the standard JSON response format.

I have a working implementation at evgfitil/fluent-bit#1. Happy to adapt based on maintainer feedback.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.