Support boto3 S3 transfer manager
- Vorherrschende Sprache
- Python
- Sterne
- 338
- Forks
- 147
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
[boto3](https://github.com/boto/boto3) has a few high-level S3 API calls like `upload_file`, `download_file` which depends on [s3transfer](https://github.com/boto/s3transfer) to perform multi-threaded object puts/gets to increase through put. This will result in a `SegmentNotFoundException` as the X-Ray recorder tries to capture the "real" S3 API call but it loses the context because the actual http call is in a worker thread from the thread pool.
The S3 transfer manager under the hood uses `futures.ThreadPoolExecutor` per https://github.com/boto/s3transfer/blob/develop/s3transfer/futures.py#L370 but there is no proper API on `boto3` client level to propagate context from user code. And requiring user code changes is not a good customer experience.
The SDK should somehow monkey-patch S3 transfer manager so that it automatically propagate context to all worker threads so each http outbound call is captured and attached to its parent segment or subsegment properly.
Library `django-storages` or any storage library that supports S3 as back-end and uses `boto3` certain APIs might face the same issue.
More detailed technical deep dive could be found here: https://github.com/aws/aws-xray-sdk-python/issues/4.
Beitragsleitfaden
Rechercherichtung
Beginne mit den in issue genannten boto3 S3 APIs, insbesondere upload_file und download_file, und lies anschließend s3transfer/futures.py im Bereich der referenzierten ThreadPoolExecutor-Implementierung. Verwende das verknüpfte aws-xray-sdk-python issue für die technische Untersuchung. Fertig ist die Aufgabe, wenn HTTP-Aufrufe aus Worker-Threads erfasst und ihrem übergeordneten Segment oder Subsegment zugeordnet werden, ohne Änderungen am User-Code zu erfordern.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- aws, python
- Bereich
- backend-api-design, cloud
- Issue-Typ
- Feature
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 35/100