Stackdriver output: metadata token response parses as invalid → "unable to parse token body" / "cannot retrieve oauth2 token" → logs stop being written to Cloud Logging on GKE
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
### Fluent Bit Version
Fluent Bit v4.2.0
### Environment
OS: Red Hat Enterprise Linux 8.10
Platform: Kubernetes (GKE)
Architecture: x86_64
Deployment: Container
Authentication: GCE metadata server (no google_service_credentials configured)
### Configuration
### Main Fluent Bit Configuration
[SERVICE]
Flush 1
Daemon Off
Log_Level error
Parsers_File /parsers.conf
@INCLUDE /opt/fluent-bit/config/logs-to-collect/logs-to-collect.conf
[OUTPUT]
Name stackdriver
Match *
resource k8s_container
export_to_project_id saas-dev-us-seddlp
Resource_Labels [REDACTED]
### Input Configuration
Multiple tail inputs monitoring application logs under:
/var/log/SymantecDLP/
/var/log/SymantecDLP/debug/
/var/log/SymantecDLP/async_inductor/
/var/log/OSMonitor/
Example input configuration:
[INPUT]
Name tail
Parser structured
Path /var/log/SymantecDLP/debug/CEHMetrics.log
DB /opt/fluent-bit/db/CEHMetrics
Tag [REDACTED]
Read_from_Head true
Path_Key tailed_path
Buffer_Chunk_Size 256k
Buffer_Max_Size 256k
Skip_Long_Lines On
Several other log files are monitored similarly (application_monitoring.log, CloudDetectionServer.log, MatcherCore.log, etc.).
###
EXPECTED BEHAVIOR
-----------------
- Metadata token response is parsed successfully when HTTP 200 is returned.
- On transient parse/network issues, token refresh is retried and logs continue to be written to Cloud Logging once a valid token is obtained.
### ACTUAL BEHAVIOR
---------------
- Intermittently, the metadata response body fails to parse despite HTTP 200.
- "unable to parse token body" and "cannot retrieve oauth2 token" are logged once.
- All subsequent flushes fail (retry or 401/403); no logs are written to Cloud Logging until Fluent Bit is restarted.
### Details
-----------
When using the stackdriver output on GKE with metadata-based auth (no google_service_credentials), Fluent Bit intermittently fails to obtain an OAuth2 token. The GCE metadata server returns HTTP 200, but the plugin fails to parse the response body and logs:
- [error] [output:stackdriver:stackdriver.0] unable to parse token body
- [error] [output:stackdriver:stackdriver.0] cannot retrieve oauth2 token
After this, no logs are successfully written to Cloud Logging. The process keeps running and the tail input keeps reading, but every flush either returns FLB_RETRY (token failure) or fails with 401 (ACCESS_TOKEN_TYPE_UNSUPPORTED) or 403 (PERMISSION_DENIED) from the Cloud Logging API. The failure does not self-recover; restart is required.
The error above is logged only once per incident; subsequent failures appear as 401/403 from the API, not as repeated "cannot retrieve oauth2 token".
ROOT CAUSE (from code and logs)
-------------------------------
- Token is fetched from http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token.
- The metadata server responds with HTTP 200, but flb_oauth2_parse_json_response() fails on the response body (in plugins/out_stackdriver/gce_metadata.c, gce_metadata_read_token()).
- So the failure is not "metadata unreachable"; it is "metadata returns 200 but the token response body cannot be parsed" (wrong/corrupt or non-token-JSON body, or client-side parsing/buffer handling).
This aligns with known reports (e.g. fluent/fluent-bit#8233) about malformed metadata requests or token handling.
STEPS TO REPRODUCE
------------------
1. Run Fluent Bit on GKE with the stackdriver output.
2. Use metadata-based auth (do not set google_service_credentials).
3. Configure stackdriver with resource k8s_container and export_to_project_id (or equivalent).
4. Let the pipeline run; the issue is intermittent (e.g. during token refresh).
5. Observe: one occurrence of "unable to parse token body" / "cannot retrieve oauth2 token", then only 401/403 from Cloud Logging and no logs delivered until restart.
RELEVANT CODE
-------------
- Error logged: plugins/out_stackdriver/stackdriver.c (~line 2918-2920), when get_google_token() returns NULL.
- Token fetch/parse: plugins/out_stackdriver/gce_metadata.c — gce_metadata_read_token(): HTTP GET to metadata, then flb_oauth2_parse_json_response(payload, ...); parse failure triggers "unable to parse token body" and "cannot retrieve oauth2 token".
EVIDENCE / LOGS
---------------
- Debug logs show: immediately before the failure, HTTP Status=200 (metadata token request), then "unable to parse token body" and "cannot retrieve oauth2 token".
- After that: only 401 (ACCESS_TOKEN_TYPE_UNSUPPORTED) and 403 (PERMISSION_DENIED) from Cloud Logging; no HTTP 200 for write requests.
- Timeline (UTC): working until ~07:17:47; first failure at 07:17:51; no recovery through at least 07:33:27.
RELATED
-------
- https://github.com/fluent/fluent-bit/issues/8233 (metadata token / malformed requests)
- https://github.com/fluent/fluent-bit/issues/8188 (stackdriver flush/retry issues)
LABELS / COMPONENTS (suggested)
-------------------------------
output-stackdriver, gce-metadata, oauth2, gke, bug
Contributor guide
Assessment
This issue has not been assessed yet.