open-telemetry / open-telemetry/opentelemetry-python-contrib

[boto3sqs] Processing spans are started to early

Open
#1,703 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted instrumentation
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

When calling the boto3 sqs.receive_message API the instrumenting function already pre-starts the processing spans for every received message before the receive_message call even returns.
Since the actual processing of the messages however happens later, somewhere in the user code, the processing spans are started too early.

Steps to reproduce

Run the test_receive_message unit test and debug the instrumenting receive_message function.
As can be seen there is a processing span pre-started for every received message (in the unit test these spans won't show up until when they are finished since span processor and in-memory exporter only track ended spans).

What is the expected behavior?
Processing spans are created when the actual processing of the SQS message happens.

What is the actual behavior?
All processing spans are immediately created in the instrumenting receive_message

Additional context
For the instrumentation it is probably hard, if not impossible, to determine when processing of an SQS message starts or ends as this is determined only by the user code.

A possible heuristical approach to the problem might be to try starting and ending the processing spans while the list of received messages is iterated. However, since there is no way of knowing how the list is actually used (e.g. random access by index, used in list comprehension to create a slightly modified new list, ...) this would also only work for some certain use-cases and might produce unwanted spans otherwise.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the instrumenting receive_message function in instrumentation/opentelemetry-instrumentation-boto3sqs/src/opentelemetry/instrumentation/boto3sqs/init.py and run test_receive_message in tests/test_boto3sqs_instrumentation.py. Trace when processing spans are created and ended relative to the received messages. Done means the test verifies processing-span timing that matches the actual message-processing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.