subsegments fail in global code with error: Subsegment discarded due to Lambda worker still initializing
- Dominant language
- Python
- Stars
- 338
- Forks
- 147
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Creating subsegments in Python global context (aka startup code) fails with errors:
It is an AWS best practice to "Initialize SDK clients and database connections outside of the function handler" https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html
However, since XRay will not create a subsegment in the global startup code, we have no view into this code with XRay. Without a subsegment, we also can not put_annotation() with XRay.
## Setup
* AWS Lambda
* Python 3.7
* XRay SDK for Python 2.4.2
## Repo
1. In Python code *outside* lambda_handler() write:
```python
with xray_recorder.in_subsegment('mysubseg') as subsegment:
subsegment.put_annotation('myanno1', '1')
print('my old school print')
```
2. Continue with any simple AWS Lambda Python function
3. Execute the function
## Result
Errors in the AWS Lambda console (if you used that) and/or errors in the Cloudwatch logs.
```
[WARNING] 2019-11-24T15:41:32.241Z Subsegment mysubseg discarded due to Lambda worker still initializing
my old school print
[WARNING] 2019-11-24T15:41:32.245Z No subsegment to end.
```
In XRay console, there is no subsegment for `mysubseg`. There is no annotation `myanno1`.
Only seen is the `Initialization` black box.
## Expected
Within `Initialization` is the subsegment `mysubseg` (and another other subsegments I might make) and on the `mysubseg` is the annotation `myanno1=1`.
## Workarounds
* Use print()
* Use the "discarded" error messages as an implied trace.
* Don't use global code or the AWS best practice approach. If you need similar functionality, consider putting it in a python class, have class initializer that sets class attributes, and on every call to the lambda handler, check if the class has yet been initialized.
## Notes
I think there is a related yet separate bug: XRay fails to show traces for boto calls in global code. I `patch(botocore)` yet calls to services like `boto3.session.Session().client('ssm')...` are invisible and not reported in any xray data or cloudwatch log.
Contributor guide
Assessment
This issue has not been assessed yet.