fluent / fluent/fluent-bit

Stackdriver Output When not on GCP/GCE

Open
#8,804 24 comments 10 reactions 0 assignees View on GitHub
enhancement
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

The stackdriver plugin appears to only want to use the google metadata service even if running off cloud. Reopening #5563

**To Reproduce**

- deploy fluentbit:3.0.3 daemonset on K3S cluster
- set `fluentbit.conf` as
```
[SERVICE]
Flush 5
Grace 120
Log_Level trace
#Log_File /var/log/fluentbit.log
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
storage.backlog.mem_limit 5M

[INPUT]
Name cpu
Tag my_cpu

[OUTPUT]
Name stackdriver
Match *
```

Error message
```
[2024/05/08 16:47:04] [ info] stackdriver.0
[2024/05/08 16:47:04] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2024/05/08 16:47:05] [debug] [stackdriver:stackdriver.0] created event channels: read=850 write=896
[2024/05/08 16:47:05] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[2024/05/08 16:47:05] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
[2024/05/08 16:47:05] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] failed to create metadata connection
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] can't fetch token from the metadata server
[2024/05/08 16:47:05] [ warn] [output:stackdriver:stackdriver.0] token retrieval failed
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] failed to create metadata connection
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] can't fetch project id from the metadata server
[2024/05/08 16:47:05] [error] [output] failed to initialize 'stackdriver' plugin
```

**Expected behavior**
The plugin should honour the GOOGLE_APPLICATION_CREDENTIALS environment variable and use the service account impersonation for workload identity federation from https://cloud.google.com/iam/docs/workload-identity-federation#oidc-credential-security

**Your Environment**
* Version used: fluentbit:3.0.3 (arm)
* Configuration: daemonset with Workload Identity Federation configured
* Environment name and version (e.g. Kubernetes? What version?): Raspberry Pi cluster running K3S 1.29
* Filters and plugins: stackdriver plugin

**Additional context**
I am setting up fluentbit logging to Google Cloud using workload identity federation. This would be for non-GCP non-GKE clusters to use GCP as a centralized log sink. I have tried adding the `google_service_credentials`, `project_id_key` and `export_to_project_id` keys in all variations and they have been ignored as the metadata service seems to be the only way the plugin gets the credentials.

The credential configuration file has the correct details for the KSA/GSA federation and I have changed the container from `fluentbit:3.0.3` to `gcloud-sdk:alpine` to test that the pod has connection to cloud logging and I can fetch the logs (I gave the SA logs writer and logs viewer in order to test). This is not a network or credential/federation issue but seems to be the plugin ignoring that it isn't in the cloud.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.