fluent / fluent/fluent-bit

out_stackdriver: service account authentication broken in v5.0.0

Open
#11,619 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

**Describe the bug**

The stackdriver output plugin's service account authentication is completely broken in v5.0.0. Both `google_service_credentials` config option and `GOOGLE_APPLICATION_CREDENTIALS` env var are ignored — the plugin only tries the GCE metadata server, which fails on non-GCP environments (AWS, Azure, on-prem, etc.).

This is a regression from v4.2.3, where the same config works correctly.

**To Reproduce**

Run fluent-bit v5.0.0 on any non-GCE machine with a valid GCP service account JSON key:

```ini
[INPUT]
name dummy

[OUTPUT]
name stackdriver
match *
google_service_credentials /path/to/service-account-key.json
```

Or equivalently in YAML format:
```yaml
pipeline:
inputs:
- name: dummy
outputs:
- name: stackdriver
match: "*"
google_service_credentials: /path/to/service-account-key.json
```

**Expected behavior**

fluent-bit reads the service account key from the specified file and authenticates via OAuth2 JWT, as documented in https://docs.fluentbit.io/manual/data-pipeline/outputs/stackdriver

**Actual behavior**

fluent-bit ignores the credentials file and only tries the GCE metadata server:

```
[output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[error] [output:stackdriver:stackdriver.0] error retrieving oauth2 access token
[error] [output:stackdriver:stackdriver.0] cannot retrieve oauth2 token
[warn] [engine] failed to flush chunk '...' cannot be retried
```

All log chunks are dropped after retry exhaustion.

**Comparison with v4.2.3**

Same config, same credentials file, same machine:

- **v5.0.0**: `error retrieving oauth2 access token` ❌
- **v4.2.3**: `[oauth2] access token from 'oauth2.googleapis.com:443' retrieved` ✅

**Environment**

- fluent-bit version: 5.0.0 (from `packages.fluentbit.io` apt repo, Ubuntu 24.04)
- OS: Ubuntu 24.04 (AWS EC2)
- Credential type: GCP service account key JSON (`"type": "service_account"`)

**Possible cause**

The OAuth2 core library (`flb_oauth2.h`) was significantly refactored between v4.2.3 and v5.0.0 (new `flb_oauth2_config` struct, JWT support for OpenTelemetry). The stackdriver plugin code itself was not changed, suggesting the regression is in the shared OAuth2 layer's handling of service account JWT auth.

**Workaround**

Pin fluent-bit to v4.x: `sudo apt-get install -y "fluent-bit=4.*"`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.