stackdriver adapter fails to filter metrics using resource metadata labels

The stackdriver adapter supports filtering time series via HPA’s LabelSelectors. Cloud Monitoring metrics have a data model where time series have two source of labels:

* Metric labels: These are labels that are automatically added to each value and augment it across different dimensions. For example, the Cloud Storage metric type `storage.googleapis.com/api/request_count` has two labels, `response_code` and `method`.
* Monitored Resource labels: These are labels which are not associated with the metric time series, but with the monitored resource associated with it (e.g. `k8s_pod`), and can be broken down into two:
* Resource labels: These are the labels that unequivocally identify the resource. For example, for `k8s_node` this will be the tuple (project id, location, cluster name, node name).
* Metadata labels: Which can be further categorized into _system_ and _user_ labels.
* System: GKE automatically adds labels to k8s monitored resources
* User: GKE automatically adds labels on the corresponding k8s resources as user metadata labels. For example, if a k8s Node resource has a label with key `beta.kubernetes.io/instance-type`, metrics associated with `k8s_node` resources will have that label defined as a user label.

When querying time series using the Cloud Monitoring API, filters for labels must be provided using a dedicated prefix for each label type: "metric.labels", "resource.labels", "metadata.system\_labels", "metadata.user\_labels". [HPA supports this today](https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/0ad301ff1487c9eabc550a52490117cf8548f1a3/custom-metrics-stackdriver-adapter/pkg/adapter/translator/query_builder.go#L40) by mapping HPA’s label selectors 1:1 to requests to the CM API.

For example, if a user wants HPA to scale based on an external metric, but filtering on a specific user label, they can do:

```
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: my-app
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
minReplicas: 1
maxReplicas: 5
metrics:
- type: External
external:
metric:
name: kubernetes.io|node|memory|allocatable_utilization
selector:
matchLabels:
metadata.user_labels.mylabel:
target:
type: AverageValue
averageValue: 600m

```

The issue arises with labels that have a prefix/name format. The prefix is optional and if provided it must be a valid DNS sub-domain name.

If a user metadata label uses this format, for example `cloud.google.com/gke-nodepool,` the filter would need to be encoded in HPA’s label selector as: `metadata.user_labels.cloud.google.com/gke-nodepool` which **is not** a valid label (the prefix is not a valid DNS sub-domain name as it contains an underscore character).

Note that this applies to both system and user labels.

**Workaround**

As a workaround, users can add their own labels to the k8s resources associated with the metric using label keys that don’t have a prefix (e.g. `gke-nodepool` instead of `cloud.google.com/gke-nodepool`. Then the resulting label key in the selector will be valid.

**Possible Fix**

We could introduce another way to filter by metadata labels by allowing two new prefixes (e.g. `metadata.system-labels` and `metadata.user-labels`). The adapter will need to special-case the handling of these labels and translate them into the prefix that Cloud Monitoring API expects.