aws / aws/aws-xray-sdk-python

subsegments fail in global code with error: Subsegment discarded due to Lambda worker still initializing

Open
#185 11 comments 13 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
338
Forks
147
PR merge metrics
No merged PRs in 30d

Description

## Summary

Creating subsegments in Python global context (aka startup code) fails with errors:

It is an AWS best practice to "Initialize SDK clients and database connections outside of the function handler" https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

However, since XRay will not create a subsegment in the global startup code, we have no view into this code with XRay. Without a subsegment, we also can not put_annotation() with XRay.

## Setup

* AWS Lambda
* Python 3.7
* XRay SDK for Python 2.4.2

## Repo

1. In Python code *outside* lambda_handler() write:

```python
with xray_recorder.in_subsegment('mysubseg') as subsegment:
subsegment.put_annotation('myanno1', '1')
print('my old school print')
```

2. Continue with any simple AWS Lambda Python function
3. Execute the function

## Result

Errors in the AWS Lambda console (if you used that) and/or errors in the Cloudwatch logs.

```
[WARNING] 2019-11-24T15:41:32.241Z Subsegment mysubseg discarded due to Lambda worker still initializing
my old school print
[WARNING] 2019-11-24T15:41:32.245Z No subsegment to end.
```

In XRay console, there is no subsegment for `mysubseg`. There is no annotation `myanno1`.
Only seen is the `Initialization` black box.

## Expected

Within `Initialization` is the subsegment `mysubseg` (and another other subsegments I might make) and on the `mysubseg` is the annotation `myanno1=1`.

## Workarounds

* Use print()
* Use the "discarded" error messages as an implied trace.
* Don't use global code or the AWS best practice approach. If you need similar functionality, consider putting it in a python class, have class initializer that sets class attributes, and on every call to the lambda handler, check if the class has yet been initialized.

## Notes

I think there is a related yet separate bug: XRay fails to show traces for boto calls in global code. I `patch(botocore)` yet calls to services like `boto3.session.Session().client('ssm')...` are invisible and not reported in any xray data or cloudwatch log.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.