expose configuration of boto retries

Open
#541 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
cloud

Research direction

Start with the retries argument in s3fs/core.py around the referenced line, then compare it with botocore's Config retry settings and the linked discussion. Determine how boto retry configuration should be exposed and how it relates to the existing retries value; done means the requested configuration is available without ambiguity.

Written by the indexing model from the issue text.

Description

This stems from https://github.com/pydata/xarray/discussions/5869#discussioncomment-1482733

See https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore-config https://github.com/boto/botocore/blob/develop/botocore/config.py#L127-L151
"""
retries (dict) --
A dictionary for retry specific configurations. Valid keys are:

'max_attempts' -- An integer representing the maximum number of retry attempts that will be made on a single request. For example, setting this value to 2 will result in the request being retried at most two times after the initial request. Setting this value to 0 will result in no retries ever being attempted on the initial request. If not provided, the number of retries will default to whatever is modeled, which is typically four retries.
"""

There is a retries arg set currently set at 5 - https://github.com/dask/s3fs/blob/main/s3fs/core.py#L158 - not sure if it related to boto retries config

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from fsspec/s3fs

All issues in fsspec/s3fs

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.