fsspec / fsspec/filesystem_spec

FTP cannot handle partial reads

Open
#641 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Any attempt to use an FTP-backed file object for something that reads less than the entire file fails, with the following stacktrace:

Stacktrace:

Traceback (most recent call last):
  File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/implementations/ftp.py", line 251, in _fetch_range
    callback=callback,
  File "/home/qwertystop/.pyenv/versions/3.7.4/lib/python3.7/ftplib.py", line 447, in retrbinary
    callback(data)
  File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/implementations/ftp.py", line 239, in callback
    raise TransferDone
fsspec.implementations.ftp.TransferDone

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/implementations/ftp.py", line 256, in _fetch_range
    self.fs.ftp.abort()
  File "/home/qwertystop/.pyenv/versions/3.7.4/lib/python3.7/ftplib.py", line 267, in abort
    raise error_proto(resp)
ftplib.error_proto: 150 0.193 seconds (measured here), 1.29 Mbytes per second

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sync_db.py", line 412, in <module>
    sync()
  File "sync_db.py", line 408, in sync
    send_to_db(files, tablename, colspec, conf["psql"], args.init, args.product)
  File "sync_db.py", line 299, in send_to_db
    header = next(reader)
  File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/spec.py", line 1449, in read
    out = self.cache._fetch(self.loc, self.loc + length)
  File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/caching.py", line 153, in _fetch
    self.cache = self.fetcher(start, end)  # new block replaces old
  File "/home/qwertystop/.pyenv/versions/hmcc-invdb/lib/python3.7/site-packages/fsspec/implementations/ftp.py", line 259, in _fetch_range
    self.fs.ftp._connect()
AttributeError: 'FTP' object has no attribute '_connect'

I initially suspected this to be a simple typo; changing self.fs.ftp._connect() to self.fs.connect() at line 259 of ftp.py is at least sufficient to prevent this specific error. However, at that point, the connection attempt reliably times out when I attempt to read an FTPFile through csv.reader, so I am unsure whether the issue is more complicated than that; I don't know enough of the FTP spec to judge that.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec/implementations/ftp.py, especially _fetch_range and its callback, then reproduce the partial-read path through fsspec/spec.py and csv.reader. Check how ftplib handles abort and reconnect after a ranged transfer. Done means an FTP-backed file can read less than the full file without the _connect error or a timeout, with regression coverage if the existing tests provide a suitable place.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.