GoogleCloudPlatform / GoogleCloudPlatform/opentelemetry-operations-python

CloudTraceSpanExporter+SimpleSpanProcessor+RequestsInstrumentor Infinite Loop

Open
#230 3 comments 0 reactions 1 assignee Claimed by @aabmass View on GitHub
bug priority: p3
Dominant language
Python
Stars
83
Forks
46
Avg merge
11h 40m
Merged PRs (30d)
2

Description

If you enable RequestsInstrumentor while using CloudTraceSpanExporter with SimpleSpanProcessor, you get an infinite loop on the first span exported.

When running locally, I see the following lines repeated with log level debug:

> I0206 10:03:10.286911 140694698043136 connectionpool.py:460] https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
> I0206 10:03:10.290570 140694689650432 requests.py:192] Making request: POST https://oauth2.googleapis.com/token

When running in cloud run, I see the following log entry repeated:

> http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 615

Because of this I assume the issue is that a requests call is made during cloud trace export, which results in a span being recorded by the instrumentor and thus a new call to cloud trace export before the first one has complete auth set-up, resulting in new auth set-up and so on.

This blog post appears to document the same issue: https://minherz.medium.com/today-i-learned-why-using-opentelemetry-requestsinstrumentor-can-freeze-your-application-ae09410b016d

Because Cloud Run requires SimpleSpanProcessor, this essentially means RequestInstrumentor cannot be used in that environment.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.