GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver

custom-metrics-sd-adapter: 403s when fetching metrics from another project

Open
#481 1 comment 1 reaction 0 assignees View on GitHub
custom-metrics-stackdriver-adapter
Dominant language
Go
Stars
409
Forks
236
Avg merge
2h 34m
Merged PRs (30d)
9

Description

I got the following error when using the credentials of the GCP service account from a project different than where `custom-metrics-stackdriver-adapter` runs.

```
provider.go:271] Failed request to stackdriver api: googleapi: Error 403: Permission monitoring.metricDescriptors.list denied (or the resource may not exist)., forbidden
```

I've confirmed that `GOOGLE_APPLICATION_CREDENTIALS` is set correctly in the container and the service account has enough permissions to read metrics. The way I validated them was to swap the `custom-metrics-stackdriver-adapter` image with [google/cloud-sdk](https://hub.docker.com/r/google/cloud-sdk/) and successfully got a list of metrics back from within an interactive shell in the container

```bash
# metrics-and-alarms is the project ID where metrics live

curl -H "Authorization: Bearer \"$(gcloud auth application-default print-access-token)\"" https://monitoring.googleapis.com/v3/projects/metrics-and-alarms/metricDescriptors
```

Here's the Kubernetes template I use. Did I mis-config anything?

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: custom-metrics-stackdriver-adapter-a1226
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: custom-metrics-stackdriver-adapter
app.kubernetes.io/version: v0.12.2-gke.0
argocd.argoproj.io/instance: custom-metrics-stackdriver-adapter-a1226
helm.sh/chart: custom-metrics-stackdriver-adapter-0.2.0
name: custom-metrics-stackdriver-adapter-a1226
namespace: custom-metrics
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/instance: custom-metrics-stackdriver-adapter-a1226
app.kubernetes.io/name: custom-metrics-stackdriver-adapter
template:
metadata:
labels:
app.kubernetes.io/instance: custom-metrics-stackdriver-adapter-a1226
app.kubernetes.io/name: custom-metrics-stackdriver-adapter
spec:
containers:
- command:
- /adapter
- '--use-new-resource-model=true'
- '--fallback-for-container-metrics=true'
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /etc/creds/token.json
image: 'gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.2-gke.0'
imagePullPolicy: IfNotPresent
name: custom-metrics-stackdriver-adapter
volumeMounts:
- mountPath: /etc/creds
name: gcp-service-account-creds
readOnly: true
serviceAccountName: custom-metrics-stackdriver-adapter-a1226
volumes:
- name: gcp-service-account-creds
secret:
defaultMode: 420
secretName: custom-metrics-stackdriver-adapter
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.