fsspec / fsspec/filesystem_spec
Skip fetch_range when you've reached the end of the file?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Is this a reasonable change?
diff --git a/fsspec/spec.py b/fsspec/spec.py
index a582e8c..54fce1a 100644
--- a/fsspec/spec.py
+++ b/fsspec/spec.py
@@ -1056,6 +1056,9 @@ class AbstractBufferedFile(io.IOBase):
length = self.size - self.loc
if self.closed:
raise ValueError('I/O operation on closed file.')
+ if self.size == self.loc and length > 0:
+ logger.debug("reached end of file. size=%d, length=%d", self.size, length)
+ return b''
logger.debug("%s read: %i - %i" % (self, self.loc, self.loc + length))
out = self.cache._fetch(self.loc, self.loc + length)
self.loc += len(out)
I noticed that gzip makes a number of read calls at the end of the file.
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 0 - 2
DEBUG:s3fs:Fetch: it-3906/data.csv.gz, 0-22161
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 2 - 10
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 10 - 11
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 11 - 12
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 12 - 13
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 13 - 14
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 14 - 15
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 15 - 16
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 16 - 17
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 17 - 18
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 18 - 19
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 19 - 8211
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 8211 - 16403
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 16403 - 24595
DEBUG:s3fs:Fetch: it-3906/data.csv.gz, 22161-24595
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 22161 - 22162
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 22161 - 22163
DEBUG:fsspec:<File-like object S3FileSystem, it-3906/data.csv.gz> read: 22161 - 22163
I suppose we don't end up making calls to (say) S3 for each of those, so maybe it's not a big deal.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fsspec/spec.py at AbstractBufferedFile.read, where the issue's proposed change checks whether the file position has reached its size. Reproduce the gzip read pattern described in the issue and verify that end-of-file reads return empty bytes without another fetch_range call, while normal reads and closed-file behavior remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100