NCAS-CMS / NCAS-CMS/PyActiveStorage
Efficiency in dask applications.
@bnlawrence is already working on this.
Since Jul 23, 2024.
- Dominant language
- Python
- Stars
- 6
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
It is clear we want to avoid pyactivestorage doing a file open inside each dask chunk if such a file open requires a remote index read each and every time.
A quick hack to fix this (in the pyfive branch) would be to avoid keeping the File instance open (the optimal_kerchunk branch already does this). With that one change, users could at least use active storage instances many times without worrying about the file open count.
A better solution long term may involve lifting the internal s3fs outside so we can take advantage of the s3fs caching.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.