Export Resource Metadata Labels

Background

My company has an internal monitoring/alerting solution that serves both our on-premise
and cloud needs; it's backed by Prometheus and Grafana. We get Cloud Monitoring (née
Stackdriver) metrics into our internal solution using this exporter.

I'm currently building some Grafana graphs for Dataflow jobs.

• Metric types under the dataflow.googleapis.com
  namespace can be filtered down to a specific Dataflow job using the job_name label
  since this label's value includes the Dataflow job name. (Technically, the job_name
  label is attached to the dataflow_job
  monitored resource type).
• Metric types under the compute.googleapis.com
  namespace can be filtered down to a specific Dataflow job using the instance_name
  label since this label's value includes the Dataflow job name.
• Metric types under the agent.googleapis.com
  namespace CANNOT be filtered down to a specific Dataflow job. None of the labels
  include the Dataflow job name. The closest label that gives me what I need is the
  instance_id label that comes attached to the gce_instance monitored resource type.
  instance_id is numeric and there's no way to know what Dataflow job it belongs to.

Because I can't filter down to a specific Dataflow job with the agent.googleapis.com
metric types, I can't build a complete Grafana dashboard that visualizes all the metrics
I care about. Because of this, I still have to go back to using a combination of
Dataflow's Job Metrics UI and Cloud Monitoring's Metrics Explorer UI just to visualize
those metrics alongside others. I'm taking on the overhead of context-switching between
3 different monitoring tools. This slows down any analysis work that I want to do.

Request

As I understand it, in addition to the labels that are attached to metric types and
monitored resource types, Cloud Monitoring also attaches
resource metadata labels
to the monitored resource types. Specifically, the metadata.system_labels.name label
includes the GCE instance name which also includes the Dataflow job name. I would be
able to use this label to filter the agent.googleapis.com metric types down to a
specific Dataflow job.

The Stackdriver exporter doesn't seem to export these resource metadata labels though.

Is there a way we can start exporting these labels by default? Or, add a configuration
that allows users to enable collection of these extra labels?

References:

• LOC where this feature might need to be implemented
• These issues look related, but not exactly the same (let me know if I'm wrong though): https://github.com/prometheus-community/stackdriver_exporter/issues/78, https://github.com/grafana/grafana/issues/18750, https://github.com/grafana/grafana/issues/18887

Happy to be corrected on anything stated here.

Start in collectors/monitoring_collector.go around line 236, the location identified by the issue, and inspect how monitored-resource labels are collected. Determine how resource metadata labels could be exported by default or enabled through configuration; done means the requested labels are available for the relevant metrics without relying on external monitoring tools.