fsspec / fsspec/filesystem_spec
make `fsspec.asyn._get_batch_size()` public
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Being able to get the configured or system default batch size from fsspec is useful when using filesystems that support an additional level of concurrency (beyond the existing fs methods that support file batching like get()/put()).
See: adlfs, which supports setting concurrency for chunked/multipart uploads and downloads (and in this case concurrency is handled by sending a batch size to the underlying Azure SDK, scheduling the individual chunk upload/download in adlfs with _run_coros_in_chunks() is not an option).
_get_batch_size() is currently marked as protected (with the leading _) and internally only gets called in asyn._run_coros_in_chunks(), but it would be better if fsspec exposed a public method for getting the configured batch size
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read fsspec.asyn._get_batch_size() and its existing caller, _run_coros_in_chunks(), first. Confirm how configured and system-default values are obtained, then verify that filesystem implementations such as adlfs can access the batch size through a public fsspec entry point.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100