open-telemetry / open-telemetry/opentelemetry-python-contrib

boto3 invoke method can not be instrumented

Open
#785 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

In case of lambda function invoking other function there is a problem with instrumentation of the invoke() method.

Describe your environment

  • https://github.com/open-telemetry/opentelemetry-lambda/tree/main/python
  • AWS Lambda Python 3.8 runtime
  • opentelemetry-instrumentation 0.24b0 (tested also on 0.25b2)
  • opentelemetry-instrumentation-boto 0.24b0 (tested also on 0.25b2)
  • opentelemetry-instrumentation-botocore 0.24b0 (tested also on 0.25b2)
  • boto3 - tested on various versions, gives the same result

Steps to reproduce
Create two AWS Lambda functions - A and B. Function A by boto3.invoke() function B.
Please see example code in the Additional context

What is the expected behavior?
I expect correct instrumentation of the boto3 invoke method and valid headers.

What is the actual behavior?

[ERROR] AttributeError: 'str' object has no attribute 'get'
Traceback (most recent call last):
  File "/opt/python/opentelemetry/instrumentation/aws_lambda/__init__.py", line 222, in _instrumented_lambda_handler_call
    result = call_wrapped(*args, **kwargs)
  File "/var/task/check_sweets.py", line 24, in get_sweets
    response = invoke(body)
  File "/var/task/check_sweets.py", line 12, in invoke
    response = client.invoke(
  File "/var/task/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/python/opentelemetry/instrumentation/botocore/__init__.py", line 215, in _patched_api_call
    BotocoreInstrumentor._patch_lambda_invoke(call_context.params)
  File "/opt/python/opentelemetry/instrumentation/botocore/__init__.py", line 178, in _patch_lambda_invoke
    headers = payload.get("headers", 
{}
)

Additional context
example code

import json
import os
import boto3

FUNCTION_NAME = os.getenv('INVOKE_FUNCTION_NAME')

client = boto3.client('lambda')


def invoke(sweets):
    print('Invoke %s with data: %s' % (FUNCTION_NAME, sweets))
    response = client.invoke(
        FunctionName=FUNCTION_NAME,
        InvocationType='RequestResponse',
        Payload=json.dumps(sweets),
    )
    return response


def get_sweets(event, context):
    body = event['body']

    print('Check if %s is available' % str(body))
    response = invoke(body)
    payload = json.loads(response["Payload"].read())

    return {'statusCode': payload['statusCode'], 'body': json.dumps(payload['body'])}

Note
The same issue created under https://github.com/open-telemetry/opentelemetry-lambda/issues/166

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with opentelemetry/instrumentation/botocore/init.py, especially _patched_api_call and _patch_lambda_invoke, and compare the reported Payload value with the expected structure. Reproduce the AWS Lambda-to-Lambda boto3 invoke example and verify that instrumentation completes without AttributeError and produces valid headers; check the referenced duplicate issue 166 for current status.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.