Allow configuring max_pool_connections on boto3 s3 client
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
🚀 Feature Request
Uploading to s3 is running out of pool connections, and the uplink is underutilized
Motivation
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: <....>.s3.amazonaws.com. Connection pool size: 10
This error keeps happing during my data processing job.
[Optional] Implementation
The key is to add max_pool_connectino config in https://github.com/mosaicml/streaming/blob/223be8f110ddeda921d8420d673d7da4d556c2c5/streaming/base/storage/upload.py#L247.
We can have a BOTO_MAX_POOL_CONNECTIONS environment variable or pass that in from the MDSWriter
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at streaming/base/storage/upload.py around line 247 and trace how the boto3 S3 client is created. Check MDSWriter's configuration path; done means callers can set max_pool_connections through the proposed environment variable or MDSWriter input, and uploads use that value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100