open-telemetry / open-telemetry/opentelemetry-python-contrib

Django with sql commentor functionality duplicates SQL queries in debug mode and CaptureQueriesContext

Open
#1,554 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

Describe your environment

Django 3.2, Python 3.10

Steps to reproduce

When instrumentation is active with either DEBUG = True turned on in settings.py or the CaptureQueriesContext context manager, it appears that multiple queries are executed. Here's a trimmed down test that shows what I mean:

def test_has_otel_comment(client: django.test.Client, span_capture: InMemorySpanExporter) -> None:
    with CaptureQueriesContext(connection) as ctx:
        response = client.get(
            "/data",
        )
    (django_span,) = span_capture.get_finished_spans()  # type: ignore[no-untyped-call]
    traceparent = otel_traceparent(django_span)

    query, _debug_query = ctx.captured_queries
    assert (
        'SELECT COUNT(*) AS "__count" FROM "auth_user"'
        f" /*controller='tests.tracing.contrib.swdjango.test_sqlcomments.data',db_driver='django.db.backends.sqlite3',framework='django%%3A{djangoversion}',route='data',traceparent='{traceparent}'*/"
        in query
    )
    assert response.json() == {"numusers": 0}

Note that the ctx.captured_queries has 2 queries - the first one is a string and is manually added by the sql_commentor middleware:

https://github.com/open-telemetry/opentelemetry-python-contrib/blob/41438ba9d20d137af0c822cf479f7326d8fe9fe8/instrumentation/opentelemetry-instrumentation-django/src/opentelemetry/instrumentation/django/middleware/sqlcommenter_middleware.py#L115-L117

This is a really strange choice to me as it makes it look like 2x the number of queries are executed. Plus, the list ends up with different shapes - Django's CursorDebugWrapper uses a dict shape for the queries log but the sqlcommentor middleware adds a separate query as a plain string.

What is the expected behavior?

A single query from CaptureQueriesContext

In my case I use the CaptureQueriesContext to count the number of queries executed to ensure that data is appropriately prefetched/select_related.

What is the actual behavior?
Multiple queries are added to the queries_log for the connection

Additional context

Researching this a bit, it looks like this was present in the original SQLCommentor implementation. I can't find information as to why though. Based on what I can see from Django, the CursorDebugWrapper reaches out through the cursor to get the raw database query that was executed against the db. This usually includes the comment with the traceparent.

Current status of the raw query in the sql key of the queries_log from the CursorDebugWrapper

EDIT - it looks like the above raw sql queries is only relevant with execute(), not executemany(). In the case of executemany(), Django intentionally disables using the cursor to retrieve the last executed query.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with sqlcommenter_middleware.py around lines 115-117 and reproduce the provided CaptureQueriesContext case, then compare its entries with Django's CursorDebugWrapper behavior for execute() across SQLite and PostgreSQL. Done means the context contains one query per execution with a consistent query-log shape, while the SQL comment remains available where supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python
Domain
backend, databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.