fsspec / fsspec/filesystem_spec
FTP cannot handle partial reads
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Any attempt to use an FTP-backed file object for something that reads less than the entire file fails, with the following stacktrace:
Stacktrace:
Traceback (most recent call last):
File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/implementations/ftp.py", line 251, in _fetch_range
callback=callback,
File "/home/qwertystop/.pyenv/versions/3.7.4/lib/python3.7/ftplib.py", line 447, in retrbinary
callback(data)
File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/implementations/ftp.py", line 239, in callback
raise TransferDone
fsspec.implementations.ftp.TransferDone
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/implementations/ftp.py", line 256, in _fetch_range
self.fs.ftp.abort()
File "/home/qwertystop/.pyenv/versions/3.7.4/lib/python3.7/ftplib.py", line 267, in abort
raise error_proto(resp)
ftplib.error_proto: 150 0.193 seconds (measured here), 1.29 Mbytes per second
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "sync_db.py", line 412, in <module>
sync()
File "sync_db.py", line 408, in sync
send_to_db(files, tablename, colspec, conf["psql"], args.init, args.product)
File "sync_db.py", line 299, in send_to_db
header = next(reader)
File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/spec.py", line 1449, in read
out = self.cache._fetch(self.loc, self.loc + length)
File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/caching.py", line 153, in _fetch
self.cache = self.fetcher(start, end) # new block replaces old
File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/implementations/ftp.py", line 259, in _fetch_range
self.fs.ftp._connect()
AttributeError: 'FTP' object has no attribute '_connect'
I initially suspected this to be a simple typo; changing self.fs.ftp._connect() to self.fs.connect() at line 259 of ftp.py is at least sufficient to prevent this specific error. However, at that point, the connection attempt reliably times out when I attempt to read an FTPFile through csv.reader, so I am unsure whether the issue is more complicated than that; I don't know enough of the FTP spec to judge that.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fsspec/implementations/ftp.py, especially _fetch_range and its callback, then reproduce the partial-read path through fsspec/spec.py and csv.reader. Check how ftplib handles abort and reconnect after a ranged transfer. Done means an FTP-backed file can read less than the full file without the _connect error or a timeout, with regression coverage if the existing tests provide a suitable place.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100