aws / aws/aws-xray-sdk-python

Custom emitter based on boto3 creates an infinite loop in the SDK

Offen
#379 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
338
Forks
147
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Hello,

I have a requirement for instrumenting a python application (specifically, an AWS Glue ETL application) using AWS X-Ray SDK but cannot have the X-Ray daemon running. Consequently, I must provide a custom emitter that can stand in place of the default `UDPEmitter`. I can go into more details of the implementation as required, the gist is that the implementation works as intended until `patch_all` method is called. The patcher will patch up the botocore and other lower level libraries, more pertinently `httplib`.

After having run the patcher, when `PutTraceSegments` API call is attempted, the application goes into infinite loop when SSO is enabled. The infinity manifests more specifically after having made a call to `GetRoleCredentials` API. Although not tested, I suspect this condition will occur even if SSO is not enabled and other types of credentials are used.

Goes something like below (my emitter is called HttpEmitter):

```text
AWSXRayRecorder.capture() -> AWSXRayRecorder.record_subsegment() -> HttpEmitter.send_entity() -> GetRoleCredentials (API) -> AWSXRayRecorder.capture() -> AWSXRayRecorder.record_subsegment() -> HttpEmitter.send_entity() ...
```

Referencing the patcher module for botocore at `aws_xray_sdk/ext/botocore/patch.py`, the issue is resolved by excluding `GetRoleCredentials` from tracing. In other words, if `GetRoleCredentials` is excluded from patching, the custom emitter based on boto3 works as expected.

```python
def _xray_traced_botocore(wrapped, instance, args, kwargs):
service = instance._service_model.metadata["endpointPrefix"]
if service == 'xray':
# skip tracing for SDK built-in sampling pollers
if ('GetSamplingRules' in args or
'GetSamplingTargets' in args or
'PutTraceSegments' in args):
return wrapped(*args, **kwargs)

if 'GetRoleCredentials' in args:
return wrapped(*args, **kwargs)

return xray_recorder.record_subsegment(
wrapped, instance, args, kwargs,
name=service,
namespace='aws',
meta_processor=aws_meta_processor,
)

```

Specifcally, this `if` is added:

```python

if 'GetRoleCredentials' in args:
return wrapped(*args, **kwargs)

```

However, I am not certain if this is in fact the correct solution or symptomatic of a more fundamental problem elsewhere, hence the ticket. I say this because the infinite loop can be made to appear just by patching `httplib` alone but it may well be for the same reason as above. In order to move forward, I have, for the moment, replaced the patcher for botocore with a custom implementation that includes the shown exclusion.

Further guidance is appreciated.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit, die Endlosschleife mit patch_all, einem benutzerdefinierten boto3-basierten Emitter und SSO zu reproduzieren. Untersuche anschließend aws_xray_sdk/ext/botocore/patch.py und den httplib-Patcher. Verfolge die Aufrufe von GetRoleCredentials und PutTraceSegments, um zu ermitteln, warum Emitter-Anfragen rekursiv getraced werden. Erledigt ist die Aufgabe, wenn der benutzerdefinierte Emitter ohne Endlosschleife funktioniert und das beabsichtigte botocore- und httplib-Tracing weiterhin funktionsfähig ist.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
aws, python
Bereich
cloud, observability-sre
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.