open-telemetry / open-telemetry/opentelemetry-python-contrib

`requests` instrumentation causes out of memory errors on Google Cloud Run

Open
#730 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

Environment
I'm working with a Python (3.8) gRPC server deployed to Cloud Run. I've installed the OpenTelemetry basics & added the gRPC and Redis instrumentation from this repo, but am having issues when adding the requests instrumentation. I've installed the following OpenTelemetry packages:

opentelemetry-api==1.5.0
opentelemetry-exporter-gcp-trace==1.0.0
opentelemetry-instrumentation==0.24b0
opentelemetry-instrumentation-grpc==0.24b0
opentelemetry-instrumentation-redis==0.24b0
opentelemetry-instrumentation-requests==0.24b0
opentelemetry-propagator-gcp==1.0.0
opentelemetry-sdk==1.5.0
opentelemetry-semantic-conventions==0.24b0
opentelemetry-util-http==0.24b0

Steps to reproduce

  1. Install opentelemetry-instrumentation-requests
  2. Call RequestsInstrumentor().instrument() when instrumenting application
  3. Deploy to Google Cloud Run

What is the expected behavior?
I would expect instrumenting the requests module to result in normal deployment & traced network requests.

What is the actual behavior?
The container running the server exceeds memory limits during deploy and fails to start. I've tried increasing the configured memory limit, but the issue persisted. Normal memory usage of this container is ~160-170MB, and limits of 256MB and 512MB have both resulted in failed deployment.

Additional context
This problem does not happen locally - checking docker stats shows expected memory usage. Additionally, I've been unable to find any reports of similar issues, however I don't know what would be incorrect in this configuration - I've basically just followed the example in the docs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the listed OpenTelemetry package versions and the RequestsInstrumentor().instrument() call, then reproduce the deployment on Google Cloud Run while comparing memory with and without requests instrumentation. Use the requests instrumentation documentation and docker stats observations as the initial reference. Done means the cause of the Cloud Run memory failure is identified and a verified fix or limitation is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, grpc, python, redis
Domain
backend, cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.