Custom emitter based on boto3 creates an infinite loop in the SDK
- Dominant language
- Python
- Stars
- 338
- Forks
- 147
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I have a requirement for instrumenting a python application (specifically, an AWS Glue ETL application) using AWS X-Ray SDK but cannot have the X-Ray daemon running. Consequently, I must provide a custom emitter that can stand in place of the default `UDPEmitter`. I can go into more details of the implementation as required, the gist is that the implementation works as intended until `patch_all` method is called. The patcher will patch up the botocore and other lower level libraries, more pertinently `httplib`.
After having run the patcher, when `PutTraceSegments` API call is attempted, the application goes into infinite loop when SSO is enabled. The infinity manifests more specifically after having made a call to `GetRoleCredentials` API. Although not tested, I suspect this condition will occur even if SSO is not enabled and other types of credentials are used.
Goes something like below (my emitter is called HttpEmitter):
```text
AWSXRayRecorder.capture() -> AWSXRayRecorder.record_subsegment() -> HttpEmitter.send_entity() -> GetRoleCredentials (API) -> AWSXRayRecorder.capture() -> AWSXRayRecorder.record_subsegment() -> HttpEmitter.send_entity() ...
```
Referencing the patcher module for botocore at `aws_xray_sdk/ext/botocore/patch.py`, the issue is resolved by excluding `GetRoleCredentials` from tracing. In other words, if `GetRoleCredentials` is excluded from patching, the custom emitter based on boto3 works as expected.
```python
def _xray_traced_botocore(wrapped, instance, args, kwargs):
service = instance._service_model.metadata["endpointPrefix"]
if service == 'xray':
# skip tracing for SDK built-in sampling pollers
if ('GetSamplingRules' in args or
'GetSamplingTargets' in args or
'PutTraceSegments' in args):
return wrapped(*args, **kwargs)
if 'GetRoleCredentials' in args:
return wrapped(*args, **kwargs)
return xray_recorder.record_subsegment(
wrapped, instance, args, kwargs,
name=service,
namespace='aws',
meta_processor=aws_meta_processor,
)
```
Specifcally, this `if` is added:
```python
if 'GetRoleCredentials' in args:
return wrapped(*args, **kwargs)
```
However, I am not certain if this is in fact the correct solution or symptomatic of a more fundamental problem elsewhere, hence the ticket. I say this because the infinite loop can be made to appear just by patching `httplib` alone but it may well be for the same reason as above. In order to move forward, I have, for the moment, replaced the patcher for botocore with a custom implementation that includes the shown exclusion.
Further guidance is appreciated.
Contributor guide
Assessment
This issue has not been assessed yet.