GoogleCloudPlatform / GoogleCloudPlatform/opentelemetry-operations-java

Metrics Exporter: Distribution |explicit_buckets.bounds| does not have at least one entry

Open
#409 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
78
Forks
39
Avg merge
14m
Merged PRs (30d)
1

Description

Metric Exporter is unable to handle Distribution metrices correctly that are collected via Micrometer instrumentation. For metrics like "jvm.gc.pause" typically the Distribtion does not have explicit bucket bounds, but the Stackdriver API is expecting at least one bound. You get a stacktrace like this:

```
com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[2] (metric.type="workload.googleapis.com/jvm.gc.pause", metric.labels={"service_name": "my-dummy-service", "instrumentation_version": "", "cause": "G1 Evacuation Pause", "instrumentation_source": "io.opentelemetry.micrometer-1.5", "gc": "G1 Young Generation", "action": "end of minor GC", "service_instance_id": "061d7963-a69f-4391-adc6-3cf5f139b13b"}): Field points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry.
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1307)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1070)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:819)
at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:568)
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:569)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1908)
at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1836)
at com.google.cloud.opentelemetry.metric.CloudMetricClientImpl.createTimeSeries(CloudMetricClientImpl.java:40)
at com.google.cloud.opentelemetry.metric.InternalMetricExporter.lambda$export$0(InternalMetricExporter.java:245)
at com.google.cloud.opentelemetry.metric.InternalMetricExporter.createTimeSeriesBatch(InternalMetricExporter.java:276)
at com.google.cloud.opentelemetry.metric.InternalMetricExporter.export(InternalMetricExporter.java:248)
at com.google.cloud.opentelemetry.metric.GoogleCloudMetricExporter.export(GoogleCloudMetricExporter.java:90)
at io.opentelemetry.sdk.metrics.export.PeriodicMetricReader$Scheduled.doRun(PeriodicMetricReader.java:167)
at io.opentelemetry.sdk.metrics.export.PeriodicMetricReader$Scheduled.run(PeriodicMetricReader.java:153)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:305)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
... 3 common frames omitted
```

In think in the implementation of the Metrics Export there is some maturity missing, especially here: https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/blob/main/exporters/metrics/src/main/java/com/google/cloud/opentelemetry/metric/MetricTranslator.java#L145

Compare it to the Stackdriver Exporter from the Micrometer project, there is lots of special handling for bucket bounds. Maybe you can learn from their implementation: https://github.com/micrometer-metrics/micrometer/blob/main/implementations/micrometer-registry-stackdriver/src/main/java/io/micrometer/stackdriver/StackdriverMeterRegistry.java#L505

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.