open-telemetry / open-telemetry/opentelemetry-python-contrib

[boto3sqs] Instrumetntation might leak memory while handling processing spans

Open
#1,704 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted instrumentation
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

The instrumentation tracks the processing spans that it creates for received SQS messages in a more or less global dictionary on the Boto3SQSInstrumentor.
As dictionary key the receipt handle of the received SQS messages is used.
Spans/entries are removed from this dictionary by the instrumentation only in one of the following cases:

According to the AWS docs the receipt handles of received messages change with every call to sqs.receive_message.
This means that the instrumentation's attempt to remove the span/entry from the dictionary in the instrumenting sqs.receive_message function is not working as intended and will leak memory if a message isn't properly deleted, e.g. if an exception happens before the sqs.delete_message API can be called.

Another case where memory might get leaked is when multiprocessing is used. E.g. SQS messages are received in a parent process and processing + deletion of messages is done in separate child processes.
The instrumentation would in the parent process then add the processing spans to the dictionary in the sqs.receive_message but when the processing spans are deleted and removed in the child process the dictionary in the parent process is unaffected.

There might be other cases but essentially if for some reason a SQS message isn't deleted in the same context as the sqs.receive_message operation, the instrumentation will leave orphaned entries its 'global' dictionary and in turn leak memory.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in instrumentation/opentelemetry-instrumentation-boto3sqs/src/opentelemetry/instrumentation/boto3sqs/init.py, especially the receive_message, delete_message, and delete_message_batch paths. Trace how processing spans are stored and removed across changed receipt handles, failed processing, and multiprocessing, then add regression coverage showing that orphaned entries do not remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.