Difference in values when comparing with Cloud Monitoring Metrics Explorer

Hi,

I am using the latest version of stackdriver_exporter to collect metrics from google cloud monitoring. However, I can see some difference in values when comparing the prometheus data with actual Metrics Explorer values in cloud monitoring.

I was checking the disk write throughput for one of my VM and the metric explorer in cloud monitoring is showing below values.

Here is the Grafana dashboard which is using prometheus data (scraped from stackdriver_exporter).

The stackdriver_exporter configuration is given below:

The prometheus is scraping the metrics from the exporter endpoint in every 30 seconds.

- job_name: gcp_metrics
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  relabel_configs:
  static_configs:
  - targets:
    - http://gcp-stackdriver-expo:31992

The cloud monitoring is showing the values above 60KiB, while the Grafana is showing less than 50 KiB. Not sure why it is showing this much difference in values. I am using the following query in Grafana to fetch data from prometheus.

sum(rate(stackdriver_gce_instance_compute_googleapis_com_instance_disk_write_bytes_count{instance_id="xxxxxxxxxxxxxxxx"}[5m])) by (instance_name)

Please correct me, if my query or any of the configuration is wrong.

Thanks

The payload names no source files or tests; begin with the stackdriver_exporter endpoint, its Google Cloud Monitoring configuration, and the Prometheus scrape and query shown. Done means reproducing the disk-write comparison and documenting whether the query or configuration explains the discrepancy.