open-telemetry / open-telemetry/opentelemetry-python
OpenTelemetry metrics exporter keeps getting configured and running with errors, no matter how hard I try to disable it
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 1k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 19
Description
Describe your environment
OS: Ubuntu 24
Python version: Python 3.12
SDK version: 1.35.0
API version: 1.35.0
What happened?
https://github.com/hiddenswitch/ComfyUI/actions/runs/16331620030/job/46135318855
I'm getting spammed with /v1/metrics errors. I don't know why these started appearing, but I don't want to configure metrics at all.
rror: -16 22:27:41 [opentelemetry.sdk.metrics._internal.export] [ERROR] [__init__.py:554] Exception while exporting metrics
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/urllib3/connection.py", line 203, in _new_conn
sock = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/usr/local/lib/python3.12/dist-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/urllib3/connectionpool.py", line 791, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/urllib3/connectionpool.py", line 497, in _make_request
conn.request(
File "/usr/local/lib/python3.12/dist-packages/urllib3/connection.py", line 395, in request
self.endheaders()
File "/usr/lib/python3.12/http/client.py", line 1331, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.12/http/client.py", line 1091, in _send_output
self.send(msg)
File "/usr/lib/python3.12/http/client.py", line 1035, in send
self.connect()
File "/usr/local/lib/python3.12/dist-packages/urllib3/connection.py", line 243, in connect
self.sock = self._new_conn()
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/urllib3/connection.py", line 218, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x748912519be0>: Failed to establish a new connection: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/urllib3/connectionpool.py", line 845, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x748912519be0>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 550, in _receive_metrics
self._exporter.export(
File "/usr/local/lib/python3.12/dist-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 339, in export
resp = self._export(serialized_data.SerializeToString())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 185, in _export
return self._session.post(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/opentelemetry/instrumentation/requests/__init__.py", line 228, in instrumented_send
return wrapped_send(self, request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/requests/adapters.py", line 700, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x748912519be0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Steps to Reproduce
if 'OTEL_METRICS_EXPORTER' not in os.environ:
os.environ['OTEL_METRICS_EXPORTER'] = 'none'
...
def _create_tracer():
resource = Resource.create({
service_attributes.SERVICE_NAME: args.otel_service_name,
service_attributes.SERVICE_VERSION: args.otel_service_version,
})
# omit progress spans from aio pika
sampler = ProgressSpanSampler()
provider = TracerProvider(resource=resource, sampler=sampler)
has_endpoint = args.otel_exporter_otlp_endpoint is not None
if has_endpoint:
otlp_exporter = OTLPSpanExporter()
else:
otlp_exporter = SpanExporter()
processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
# enable instrumentation
patch_spanbuilder_set_channel()
AioPikaInstrumentor().instrument()
AioHttpServerInstrumentor().instrument()
RequestsInstrumentor().instrument()
return trace.get_tracer(args.otel_service_name)
from https://github.com/hiddenswitch/ComfyUI/blob/master/comfy/cmd/main_pre.py
Expected Result
I don't want to export metrics... nothing I am trying is preventing metrics from being exported!
Actual Result
Spam about /v1/metrics
Additional context
I haven't changed anything about my otel setup in the test environment.
Would you like to implement a fix?
Yes
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the metrics exporter configuration path in the Python SDK and compare it with the reproduction in comfy/cmd/main_pre.py, especially the OTEL_METRICS_EXPORTER=none environment setting. Trace why the OTLP metrics exporter is still initialized and review the shown export traceback. Done means disabling metrics prevents requests to /v1/metrics and the repeated export errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100