aws / aws/aws-xray-sdk-python

Custom emitter based on boto3 creates an infinite loop in the SDK

Open
#379 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
338
Forks
147
PR merge metrics
No merged PRs in 30d

Description

Hello,

I have a requirement for instrumenting a python application (specifically, an AWS Glue ETL application) using AWS X-Ray SDK but cannot have the X-Ray daemon running. Consequently, I must provide a custom emitter that can stand in place of the default `UDPEmitter`. I can go into more details of the implementation as required, the gist is that the implementation works as intended until `patch_all` method is called. The patcher will patch up the botocore and other lower level libraries, more pertinently `httplib`.

After having run the patcher, when `PutTraceSegments` API call is attempted, the application goes into infinite loop when SSO is enabled. The infinity manifests more specifically after having made a call to `GetRoleCredentials` API. Although not tested, I suspect this condition will occur even if SSO is not enabled and other types of credentials are used.

Goes something like below (my emitter is called HttpEmitter):

```text
AWSXRayRecorder.capture() -> AWSXRayRecorder.record_subsegment() -> HttpEmitter.send_entity() -> GetRoleCredentials (API) -> AWSXRayRecorder.capture() -> AWSXRayRecorder.record_subsegment() -> HttpEmitter.send_entity() ...
```

Referencing the patcher module for botocore at `aws_xray_sdk/ext/botocore/patch.py`, the issue is resolved by excluding `GetRoleCredentials` from tracing. In other words, if `GetRoleCredentials` is excluded from patching, the custom emitter based on boto3 works as expected.

```python
def _xray_traced_botocore(wrapped, instance, args, kwargs):
service = instance._service_model.metadata["endpointPrefix"]
if service == 'xray':
# skip tracing for SDK built-in sampling pollers
if ('GetSamplingRules' in args or
'GetSamplingTargets' in args or
'PutTraceSegments' in args):
return wrapped(*args, **kwargs)

if 'GetRoleCredentials' in args:
return wrapped(*args, **kwargs)

return xray_recorder.record_subsegment(
wrapped, instance, args, kwargs,
name=service,
namespace='aws',
meta_processor=aws_meta_processor,
)

```

Specifcally, this `if` is added:

```python

if 'GetRoleCredentials' in args:
return wrapped(*args, **kwargs)

```

However, I am not certain if this is in fact the correct solution or symptomatic of a more fundamental problem elsewhere, hence the ticket. I say this because the infinite loop can be made to appear just by patching `httplib` alone but it may well be for the same reason as above. In order to move forward, I have, for the moment, replaced the patcher for botocore with a custom implementation that includes the shown exclusion.

Further guidance is appreciated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.