fsspec / fsspec/gcsfs

[Feature Request] Decouple read cache size and write buffer size in AbstractBufferedFile

Open
#761 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

The Problem
Currently, AbstractBufferedFile uses a single argument, block_size, to control both:

  1. The size of the read cache (passed to cache initialization).
  2. The write buffer size (triggering a flush/upload when the buffer exceeds this size).

This coupling creates a limitation for downstream implementations (like gcsfs) where optimal performance might require different values for reading and writing.

Code References

Motivation / Use Case
In gcsfs.ExtendedGcsFilesystem, we want to set a larger buffer for writes (e.g., 16MB default used by python SDK) to optimize upload throughput. However, for reads, we want to keep the cache size smaller (e.g., the default 5MB) to limit memory usage during random access or with many open files.

Because block_size controls both, we cannot increase the write buffer default without unintentionally increasing the read cache, nor can users configure them independently.

Proposed Solution
We propose adding a new argument to AbstractBufferedFile and GCSFileSystem, such as write_block_size (or write_buffer_size).

  • If the new argument is provided, it controls the write buffer threshold.
  • If it is None, it falls back to block_size to maintain backward compatibility.

Contribution
We are happy to submit a PR to implement this change.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fsspec/spec.py at the referenced read-cache and write-buffer locations, then trace how GCSFileSystem constructs AbstractBufferedFile. Add an optional write-size setting that falls back to block_size, and verify that reads retain their configured cache size while writes use the independent threshold without breaking existing callers.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.