open-telemetry / open-telemetry/opentelemetry-python-contrib
[boto3sqs] Processing spans are started to early
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 1.1k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 16
Description
When calling the boto3 sqs.receive_message API the instrumenting function already pre-starts the processing spans for every received message before the receive_message call even returns.
Since the actual processing of the messages however happens later, somewhere in the user code, the processing spans are started too early.
Steps to reproduce
Run the test_receive_message unit test and debug the instrumenting receive_message function.
As can be seen there is a processing span pre-started for every received message (in the unit test these spans won't show up until when they are finished since span processor and in-memory exporter only track ended spans).
What is the expected behavior?
Processing spans are created when the actual processing of the SQS message happens.
What is the actual behavior?
All processing spans are immediately created in the instrumenting receive_message
Additional context
For the instrumentation it is probably hard, if not impossible, to determine when processing of an SQS message starts or ends as this is determined only by the user code.
A possible heuristical approach to the problem might be to try starting and ending the processing spans while the list of received messages is iterated. However, since there is no way of knowing how the list is actually used (e.g. random access by index, used in list comprehension to create a slightly modified new list, ...) this would also only work for some certain use-cases and might produce unwanted spans otherwise.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the instrumenting receive_message function in instrumentation/opentelemetry-instrumentation-boto3sqs/src/opentelemetry/instrumentation/boto3sqs/init.py and run test_receive_message in tests/test_boto3sqs_instrumentation.py. Trace when processing spans are created and ended relative to the received messages. Done means the test verifies processing-span timing that matches the actual message-processing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100