`flush` doesn't create or upload the file until the file is closed
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
fsspec version 2022.5.0
gcsfs version 2022.5.0
Code to reproduce:
import fsspec
fileobj = fsspec.open('gs://<insert-your-bucket-here>/test-write-flush', 'w', auto_mkdirs=True)
f = fileobj.fs.open(fileobj.path, mode=fileobj.mode)
f.write('w' * (2**20)) # is guaranteed to be larger than minimal block size
f.flush() # does nothing visible - no file is created at the destination
f.close() # now the file is created and has content
Upon debugging flush call, it seems that the check self.buffer.tell() < self.blocksize is always True, because the way things are implemented, self.buffer.tell() returns 0.
Furthermore, if I call manually what is in fsspec flush implementation after the check, meaning this code:
if self.offset is None:
# Initialize a multipart upload
self.offset = 0
try:
self._initiate_upload()
except: # noqa: E722
self.closed = True
raise
if self._upload_chunk(final=force) is not False:
self.offset += self.buffer.seek(0, 2)
self.buffer = io.BytesIO()
the file is still not created, although the underlying code in _upload_chunk does something.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the provided fsspec/gcsfs reproduction and trace the flush implementation, especially the buffer.tell() check, _initiate_upload(), and _upload_chunk(). Confirm the expected behavior by checking that flush creates or uploads the Google Cloud Storage object before close, while preserving the reported close behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100