Inconsistent metrics from storage.googleapis.com

Hello,

I encounter some issue to grab the storage metrics. Some time I've got all the metrics, some time just some of them and some times none.

I've started it with just storage and verbose output

STACKDRIVER_EXPORTER_MONITORING_METRICS_TYPE_PREFIXES=storage.googleapis.com/

And observed the following logs:

testsd_1  | time="2018-11-15T08:27:11Z" level=debug msg="Listing Google Stackdriver Monitoring metric descriptors starting with `storage.googleapis.com/`..." source="monitoring_collector.go:213"
testsd_1  | time="2018-11-15T08:27:11Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `storage.googleapis.com/network/received_bytes_count`..." source="monitoring_collector.go:169"
testsd_1  | time="2018-11-15T08:27:11Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `storage.googleapis.com/network/sent_bytes_count`..." source="monitoring_collector.go:169"
testsd_1  | time="2018-11-15T08:27:11Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `storage.googleapis.com/storage/object_count`..." source="monitoring_collector.go:169"
testsd_1  | time="2018-11-15T08:27:11Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `storage.googleapis.com/api/request_count`..." source="monitoring_collector.go:169"
testsd_1  | time="2018-11-15T08:27:11Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `storage.googleapis.com/storage/total_byte_seconds`..." source="monitoring_collector.go:169"
testsd_1  | time="2018-11-15T08:27:11Z" level=debug msg="Retrieving Google Stackdriver Monitoring metrics for descriptor `storage.googleapis.com/storage/total_bytes`..." source="monitoring_collector.go:169"

But only one metric is returned

# curl -vs localhost:9256/metrics 2>&1 | grep gcs_bucket | sed 's/{.*/{.../'
# HELP stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count Total number of objects per bucket, grouped by storage class. Values are measured once per day.
# TYPE stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count gauge
stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count{...
stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count{...
stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count{...
stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count{...

Or two

# curl -vs localhost:9256/metrics 2>&1 | grep gcs_bucket | sed 's/{.*/{.../'
# HELP stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count Total number of objects per bucket, grouped by storage class. Values are measured once per day.
# TYPE stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count gauge
stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count{...
stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count{...
stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count{...
stackdriver_gcs_bucket_storage_googleapis_com_storage_object_count{...
# HELP stackdriver_gcs_bucket_storage_googleapis_com_storage_total_byte_seconds Total daily storage in byte*seconds used by the bucket, grouped by storage class. * Connection #0 to host localhost left intact
# TYPE stackdriver_gcs_bucket_storage_googleapis_com_storage_total_byte_seconds gauge
stackdriver_gcs_bucket_storage_googleapis_com_storage_total_byte_seconds{...
stackdriver_gcs_bucket_storage_googleapis_com_storage_total_byte_seconds{...
stackdriver_gcs_bucket_storage_googleapis_com_storage_total_byte_seconds{...
stackdriver_gcs_bucket_storage_googleapis_com_storage_total_byte_seconds{...

Any idea why all metrics (like stackdriver_gcs_bucket_storage_googleapis_com_storage_total_bytes) are not returned each time ?

Thanks.

Start in monitoring_collector.go at the descriptor listing and metric retrieval paths referenced by the debug logs. Reproduce requests for the storage.googleapis.com/ prefix and compare the descriptors and responses across runs; done means identifying why metrics are intermittently omitted and documenting or testing the corrected behavior.