fluent / fluent/fluent-bit

Stackdriver output: metadata token response parses as invalid → "unable to parse token body" / "cannot retrieve oauth2 token" → logs stop being written to Cloud Logging on GKE

Open
#11,505 17 comments 5 reactions 0 assignees View on GitHub
Stale waiting-for-user
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

### Fluent Bit Version
Fluent Bit v4.2.0

### Environment
OS: Red Hat Enterprise Linux 8.10
Platform: Kubernetes (GKE)
Architecture: x86_64
Deployment: Container
Authentication: GCE metadata server (no google_service_credentials configured)

### Configuration
### Main Fluent Bit Configuration
[SERVICE]
Flush 1
Daemon Off
Log_Level error
Parsers_File /parsers.conf

@INCLUDE /opt/fluent-bit/config/logs-to-collect/logs-to-collect.conf

[OUTPUT]
Name stackdriver
Match *
resource k8s_container
export_to_project_id saas-dev-us-seddlp
Resource_Labels [REDACTED]

### Input Configuration

Multiple tail inputs monitoring application logs under:
/var/log/SymantecDLP/
/var/log/SymantecDLP/debug/
/var/log/SymantecDLP/async_inductor/
/var/log/OSMonitor/

Example input configuration:
[INPUT]
Name tail
Parser structured
Path /var/log/SymantecDLP/debug/CEHMetrics.log
DB /opt/fluent-bit/db/CEHMetrics
Tag [REDACTED]
Read_from_Head true
Path_Key tailed_path
Buffer_Chunk_Size 256k
Buffer_Max_Size 256k
Skip_Long_Lines On
Several other log files are monitored similarly (application_monitoring.log, CloudDetectionServer.log, MatcherCore.log, etc.).

###
EXPECTED BEHAVIOR
-----------------
- Metadata token response is parsed successfully when HTTP 200 is returned.
- On transient parse/network issues, token refresh is retried and logs continue to be written to Cloud Logging once a valid token is obtained.

### ACTUAL BEHAVIOR
---------------
- Intermittently, the metadata response body fails to parse despite HTTP 200.
- "unable to parse token body" and "cannot retrieve oauth2 token" are logged once.
- All subsequent flushes fail (retry or 401/403); no logs are written to Cloud Logging until Fluent Bit is restarted.

### Details

-----------
When using the stackdriver output on GKE with metadata-based auth (no google_service_credentials), Fluent Bit intermittently fails to obtain an OAuth2 token. The GCE metadata server returns HTTP 200, but the plugin fails to parse the response body and logs:

- [error] [output:stackdriver:stackdriver.0] unable to parse token body
- [error] [output:stackdriver:stackdriver.0] cannot retrieve oauth2 token

After this, no logs are successfully written to Cloud Logging. The process keeps running and the tail input keeps reading, but every flush either returns FLB_RETRY (token failure) or fails with 401 (ACCESS_TOKEN_TYPE_UNSUPPORTED) or 403 (PERMISSION_DENIED) from the Cloud Logging API. The failure does not self-recover; restart is required.

The error above is logged only once per incident; subsequent failures appear as 401/403 from the API, not as repeated "cannot retrieve oauth2 token".

ROOT CAUSE (from code and logs)
-------------------------------
- Token is fetched from http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token.
- The metadata server responds with HTTP 200, but flb_oauth2_parse_json_response() fails on the response body (in plugins/out_stackdriver/gce_metadata.c, gce_metadata_read_token()).
- So the failure is not "metadata unreachable"; it is "metadata returns 200 but the token response body cannot be parsed" (wrong/corrupt or non-token-JSON body, or client-side parsing/buffer handling).

This aligns with known reports (e.g. fluent/fluent-bit#8233) about malformed metadata requests or token handling.

STEPS TO REPRODUCE
------------------
1. Run Fluent Bit on GKE with the stackdriver output.
2. Use metadata-based auth (do not set google_service_credentials).
3. Configure stackdriver with resource k8s_container and export_to_project_id (or equivalent).
4. Let the pipeline run; the issue is intermittent (e.g. during token refresh).
5. Observe: one occurrence of "unable to parse token body" / "cannot retrieve oauth2 token", then only 401/403 from Cloud Logging and no logs delivered until restart.

RELEVANT CODE
-------------
- Error logged: plugins/out_stackdriver/stackdriver.c (~line 2918-2920), when get_google_token() returns NULL.
- Token fetch/parse: plugins/out_stackdriver/gce_metadata.c — gce_metadata_read_token(): HTTP GET to metadata, then flb_oauth2_parse_json_response(payload, ...); parse failure triggers "unable to parse token body" and "cannot retrieve oauth2 token".

EVIDENCE / LOGS
---------------
- Debug logs show: immediately before the failure, HTTP Status=200 (metadata token request), then "unable to parse token body" and "cannot retrieve oauth2 token".
- After that: only 401 (ACCESS_TOKEN_TYPE_UNSUPPORTED) and 403 (PERMISSION_DENIED) from Cloud Logging; no HTTP 200 for write requests.
- Timeline (UTC): working until ~07:17:47; first failure at 07:17:51; no recovery through at least 07:33:27.

RELATED
-------
- https://github.com/fluent/fluent-bit/issues/8233 (metadata token / malformed requests)
- https://github.com/fluent/fluent-bit/issues/8188 (stackdriver flush/retry issues)

LABELS / COMPONENTS (suggested)
-------------------------------
output-stackdriver, gce-metadata, oauth2, gke, bug

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.