cloud-bulldozer / cloud-bulldozer/performance-dashboards

PROMETHEUS_BEARER expire

Open
#66 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jsonnet
Stars
20
Forks
46
PR merge metrics
No merged PRs in 30d

Description

After Dittybopper dashboard deployed for several hours, the dashboards got Forbidden. Delete dittybopper and deploy again can fix this issue. But after several hours, I see the same issue. When the issue is seen, curl with the token in sc-ocp-prom configmap to prometheus url also got 403. But manually creating a new token can make it work. I doubt the token expired.

```
URL=$(oc get route -n openshift-monitoring prometheus-k8s -o jsonpath="{.spec.host}")
```
Get the token in sc-ocp-prom configmap
```
% oc get cm sc-ocp-prom -n dittybopper -o jsonpath="{.metadata.annotations}"
```
The token in sc-ocp-prom configmap got 403
```
% export TOKEN=

% curl -ksl -H "Authorization: Bearer $TOKEN" https://${URL}/api/v1/status/runtimeinfo -w "%{http_code}" -o /dev/null
403%
```
Use https://jwt.io/ to decode the token token I got `"exp": 1676971045`
```
$ date -d @1676971045
Tue Feb 21 09:17:25 AM UTC 2023
```
New token works well.
```
% export TOKEN_NEW=$(oc create token -n openshift-monitoring prometheus-k8s)
% curl -k -H "Authorization: Bearer $TOKEN_NEW" https://${URL}/api/v1/status/runtimeinfo
{"status":"success","data":{"startTime":"2023-02-22T02:30:14.309517962Z","CWD":"/prometheus","reloadConfigSuccess":true,"lastConfigTime":"2023-02-22T02:30:20Z","corruptionCount":0,"goroutineCount":1723,"GOMAXPROCS":4,"GOGC":"","GODEBUG":"","storageRetention":"15d"}}%
```

I think we can use --duration to set a longer time to the token, like 240h, not sure if we can set it as forever.
oc create token prometheus-k8s -n openshift-monitoring --duration 240h

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the sc-ocp-prom ConfigMap and the prometheus-k8s token used by the dashboard, then reproduce the curl request against the openshift-monitoring Prometheus URL. Determine why the stored token returns 403 after several hours and define the token-handling change needed so the dashboard retains access; verify the request succeeds with the resulting configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.