Phantom Metrics being emitted

Cloud: AWS
Runtime: Docker
Python Version: 3.11
Otel Version: 1.20.0
Collector: Otel-collector

Steps to reproduce
Describe exactly how to reproduce the error. Include a code sample if applicable.

init.py to configure provider:

from opentelemetry import metrics
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
from interfaces.otel.resource import OtelResource
from opentelemetry.metrics._internal import _ProxyMeterProvider
from utilities import Config


if Config.IS_LOCAL_OTEL:
    exporter = ConsoleMetricExporter()
else:
    exporter = OTLPMetricExporter(endpoint=Config.OTEL_EXPORTER_OTLP_ENDPOINT)
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=Config.OTEL_METRIC_EXPORT_INTERVAL_MS)
provider = metrics.get_meter_provider()
if isinstance(provider, _ProxyMeterProvider):
  provider = MeterProvider([reader], resource=OtelResource(
                service_name=Config.OTEL_SERVICE_NAME,
                service_version=Config.OTEL_SERVICE_VERSION,
                service_environment=Config.OTEL_SERVICE_ENVIRONMENT
              ).resource)
  metrics.set_meter_provider(provider)


def force_flush():
  """
  Helper function to otel metrics force flush call
  """
  provider.force_flush()

def shutdown():
  """
  Helper function to otel metrics shutdown call
  """
  provider.shutdown()

Code usage:

from interfaces.otel.metrics import metrics

_meter = metrics.get_meter(__name__)
_retrievals_counter = _meter.create_counter("retrievals")

I see these random massive peaks in metrics being reported. By massive im talking in the order of thousands to tens of thousands. When I go and check our APM to see how many times this metric should have been emitted it is not even close to the expected metric value. Sometimes the API wouldn't have been called at all and there are still metrics emitted as if it has been hit.

What is the expected behavior?

Metrics to not be reported that should not be being reported.

What is the actual behavior?

Phantom metrics being reported in datadog.

Additional context

None

Start with the provider setup in init.py and the meter usage in interfaces.otel.metrics, then reproduce the reported behavior with the supplied Python 3.11, Docker, AWS, and collector context. Compare the counter calls with the metrics exported to Datadog; done means unsupported or unexpected emissions are explained and no phantom metrics remain.