open-telemetry / open-telemetry/opentelemetry-python

Unexpected behaviour observed after integrating with Otel

Open
#3,860 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.6k
Forks
1k
Avg merge
4d 15h
Merged PRs (30d)
19

Description

Describe your environment Describe any aspect of your environment relevant to the problem, including your Python version, platform, version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on main.

  1. Our service runs in docker.
  2. We use falcon server with gunicorn integration.
  3. We also use gevent patching for faster and non-blocking IO.
  4. We are using below versions of Otel:
  5. opentelemetry-api 1.24.0 opentelemetry-distro 0.45b0 opentelemetry-exporter-otlp 1.24.0 opentelemetry-exporter-otlp-proto-common 1.24.0 opentelemetry-exporter-otlp-proto-grpc 1.24.0 opentelemetry-exporter-otlp-proto-http 1.24.0 opentelemetry-instrumentation 0.45b0 opentelemetry-instrumentation-asgi 0.45b0 opentelemetry-instrumentation-asyncio 0.45b0 opentelemetry-instrumentation-aws-lambda 0.45b0 opentelemetry-instrumentation-boto 0.45b0 opentelemetry-instrumentation-boto3sqs 0.45b0 opentelemetry-instrumentation-botocore 0.45b0 opentelemetry-instrumentation-dbapi 0.45b0 opentelemetry-instrumentation-falcon 0.45b0 opentelemetry-instrumentation-grpc 0.45b0 opentelemetry-instrumentation-logging 0.45b0 opentelemetry-instrumentation-pymemcache 0.45b0 opentelemetry-instrumentation-pymysql 0.45b0 opentelemetry-instrumentation-requests 0.45b0 opentelemetry-instrumentation-sqlite3 0.45b0 opentelemetry-instrumentation-system-metrics 0.45b0 opentelemetry-instrumentation-urllib 0.45b0 opentelemetry-instrumentation-urllib3 0.45b0 opentelemetry-instrumentation-wsgi 0.45b0 opentelemetry-propagator-aws-xray 1.0.1 opentelemetry-proto 1.24.0 opentelemetry-sdk 1.24.0 opentelemetry-semantic-conventions 0.45b0 opentelemetry-test-utils 0.45b0 opentelemetry-util-http 0.45b0

Steps to reproduce
Describe exactly how to reproduce the error. Include a code sample if applicable.

  1. Instrument falcon server in gunicorn master before server startup.
  2. Initialize Otel in post init worker.
  3. On a very higher load we are observing below issue:
  4. Our service made a db query Q1 and query Q2 at the same time. The results of Q1 and Q2 are getting interchanged. Suspect over here is that the db connection is somehow getting shared.
    What is the expected behavior?
    What did you expect to see?
  5. No db connection sharing should happen.
    What is the actual behavior?
    What did you see instead?
  6. We are suspecting db connection sharing.
    Additional context
    Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported high-load case using Falcon with Gunicorn, gevent patching, and the listed OpenTelemetry versions, following the issue's master and worker initialization steps. Investigate whether database connections are shared between workers or concurrent requests; done means simultaneous Q1 and Q2 consistently return their own results. No repository files or tests are named.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.