apache / apache/beam

[Task]: Set the Beam user agent on the Python boto3 S3 client

Open
#39,573 1 comment 0 reactions 1 assignee Claimed by @goanpeca View on GitHub
P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What needs to happen?

`sdks/python/apache_beam/io/aws/clients/s3/boto3_client.py` constructs its boto3 client without a botocore `Config`, so S3 requests from Beam's Python AWS I/O carry only the default `Boto3/... botocore/...` user agent. Object store operators cannot attribute that traffic to Beam, which makes it harder to diagnose throughput, throttling, or request-volume questions raised by Beam users.

Beam already does this for its other cloud I/O connectors. `apache_beam/io/gcp/gcsio.py:181`, `apache_beam/io/gcp/bigquery_tools.py:365`, and `apache_beam/io/gcp/datastore/v1new/helper.py:55` all set a Beam-identifying user agent on their clients. The S3 client is the inconsistent one.

The change is to pass `config=Config(user_agent_extra='apache-beam/')` when building the client, reading the version from `apache_beam.version.__version__`. `user_agent_extra` appends to the standard botocore user agent rather than replacing it, so nothing that parses the existing string changes.

This benefits operators of Amazon S3 and of any other implementation of the S3 API equally.

I have a working branch with a unit test that asserts the version string appears in `client.meta.config.user_agent`, and can open a PR against this issue.

### Issue Priority

Priority: 3 (nice-to-have improvement)

### Issue Components

- Component: Python SDK
- Component: IO connector

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.