fluent / fluent/fluent-bit

Expose K8S API response for failure

Open
#6,551 0 comments 0 reactions 1 assignee Claimed by @patrick-stephens View on GitHub
enhancement
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

**Is your feature request related to a problem? Please describe.**

The response from the K8S API when we attempt to query metadata is only logged at `debug` level.
This means if you have a misconfiguration (e.g. RBAC does not allow us to query things with the service account) then you just get a generic "not ok" message logged.
Stepping up to debug level shows the actual response saying RBAC is not configured in this case so is incredibly helpful: in the example below you can see the response message explaining the failure immediately at `debug` level - by default you just get the `warn` message with no details.

```
[2022/12/08 15:53:38] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2022/12/08 15:53:38] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information
[2022/12/08 15:53:38] [debug] [http_client] not using http_proxy for header
[2022/12/08 15:53:38] [debug] [http_client] server kubernetes.default.svc:443 will close connection #23
[2022/12/08 15:53:38] [debug] [filter:kubernetes:kubernetes.0] Request (ns=default, pod=calyptia-cluster-logging-316c-dcr7d) http_do=0, HTTP Status: 403
[2022/12/08 15:53:38] [debug] [filter:kubernetes:kubernetes.0] HTTP response
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"calyptia-cluster-logging-316c-dcr7d\" is forbidden: User \"system:serviceaccount:default:default\" cannot get resource \"pods\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"calyptia-cluster-logging-316c-dcr7d","kind":"pods"},"code":403}

[2022/12/08 15:53:38] [ warn] [filter:kubernetes:kubernetes.0] could not get meta for POD calyptia-cluster-logging-316c-dcr7d
```

**Describe the solution you'd like**
Log the response, particularly for failure cases, at `info` or above level.

**Describe alternatives you've considered**

**Additional context**

Deploying FB into a cluster without setting up the relevant permissions for K8S API querying took a while to figure out.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.