[Feature Request] Decouple read cache size and write buffer size in AbstractBufferedFile
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
The Problem
Currently, AbstractBufferedFile uses a single argument, block_size, to control both:
- The size of the read cache (passed to cache initialization).
- The write buffer size (triggering a flush/upload when the buffer exceeds this size).
This coupling creates a limitation for downstream implementations (like gcsfs) where optimal performance might require different values for reading and writing.
Code References
- Read Context:
block_sizeis passed to initialize the read cache here:
https://github.com/fsspec/filesystem_spec/blob/e12aa7571244f6695264c92c4867978fed5ad092/fsspec/spec.py#L1919 - Write Context: The same
block_sizeis used to check if the write buffer is full here:
https://github.com/fsspec/filesystem_spec/blob/e12aa7571244f6695264c92c4867978fed5ad092/fsspec/spec.py#L2033
Motivation / Use Case
In gcsfs.ExtendedGcsFilesystem, we want to set a larger buffer for writes (e.g., 16MB default used by python SDK) to optimize upload throughput. However, for reads, we want to keep the cache size smaller (e.g., the default 5MB) to limit memory usage during random access or with many open files.
Because block_size controls both, we cannot increase the write buffer default without unintentionally increasing the read cache, nor can users configure them independently.
Proposed Solution
We propose adding a new argument to AbstractBufferedFile and GCSFileSystem, such as write_block_size (or write_buffer_size).
- If the new argument is provided, it controls the write buffer threshold.
- If it is
None, it falls back toblock_sizeto maintain backward compatibility.
Contribution
We are happy to submit a PR to implement this change.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with fsspec/spec.py at the referenced read-cache and write-buffer locations, then trace how GCSFileSystem constructs AbstractBufferedFile. Add an optional write-size setting that falls back to block_size, and verify that reads retain their configured cache size while writes use the independent threshold without breaking existing callers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100