fsspec / fsspec/gcsfs

`flush` doesn't create or upload the file until the file is closed

Open
#484 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

fsspec version 2022.5.0
gcsfs version 2022.5.0

Code to reproduce:

import fsspec
fileobj = fsspec.open('gs://<insert-your-bucket-here>/test-write-flush', 'w', auto_mkdirs=True)
f = fileobj.fs.open(fileobj.path, mode=fileobj.mode)
f.write('w' * (2**20))  # is guaranteed to be larger than minimal block size
f.flush()  # does nothing visible - no file is created at the destination
f.close()  # now the file is created and has content

Upon debugging flush call, it seems that the check self.buffer.tell() < self.blocksize is always True, because the way things are implemented, self.buffer.tell() returns 0.
Furthermore, if I call manually what is in fsspec flush implementation after the check, meaning this code:

        if self.offset is None:
            # Initialize a multipart upload
            self.offset = 0
            try:
                self._initiate_upload()
            except:  # noqa: E722
                self.closed = True
                raise

        if self._upload_chunk(final=force) is not False:
            self.offset += self.buffer.seek(0, 2)
            self.buffer = io.BytesIO()

the file is still not created, although the underlying code in _upload_chunk does something.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the provided fsspec/gcsfs reproduction and trace the flush implementation, especially the buffer.tell() check, _initiate_upload(), and _upload_chunk(). Confirm the expected behavior by checking that flush creates or uploads the Google Cloud Storage object before close, while preserving the reported close behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.