fsspec / fsspec/filesystem_spec

FTPFileSystem open file cannot read from filezilla server

Open
#1,417 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

The ABOR command is sent when the ftp file download is finished, the server no longer responds and raises the TimeoutError

code:

def test1():
    fs: FTPFileSystem = fsspec.filesystem('ftp', asynchronous=True, host='127.0.0.1', port=21, username='admin',
                                          password='admin', timeout=10)

    with fs.open(fs.ls('/realtime_update')[0]['name'], compression='gzip') as f:
        f.read()

fsspec.implementations.ftp.FTPFile._fetch_range

 # stop transfer, we got enough bytes for this block
                self.fs.ftp.abort()

debug log

*get* '150 Starting data transfer.\n'
*resp* '150 Starting data transfer.'
*put urgent* b'ABOR\r\n'
*get* '226 Operation successful\n'
Traceback (most recent call last):
  File "F:\project\python\external_ftp_pipeline\.venv\Lib\site-packages\fsspec\implementations\ftp.py", line 323, in _fetch_range
    self.fs.ftp.retrbinary(
  File "D:\Applications\scoop\apps\python\current\Lib\ftplib.py", line 441, in retrbinary
    callback(data)
  File "F:\project\python\external_ftp_pipeline\.venv\Lib\site-packages\fsspec\implementations\ftp.py", line 315, in callback
    raise TransferDone
fsspec.implementations.ftp.TransferDone

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\project\python\external_ftp_pipeline\src\external_ftp\gongshang_ftp.py", line 154, in <module>
    asyncio.run(main())
  File "D:\Applications\scoop\apps\python\current\Lib\asyncio\runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "D:\Applications\scoop\apps\python\current\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Applications\scoop\apps\python\current\Lib\asyncio\base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "F:\project\python\external_ftp_pipeline\src\external_ftp\gongshang_ftp.py", line 136, in main
    for tag, data in load_ftp_infos():
  File "F:\project\python\external_ftp_pipeline\src\external_ftp\gongshang_ftp.py", line 127, in load_ftp_infos
    for data in decode_lines(f):
  File "F:\project\python\external_ftp_pipeline\src\external_ftp\gongshang_ftp.py", line 117, in decode_lines
    for line in source:
  File "D:\Applications\scoop\apps\python\current\Lib\gzip.py", line 399, in readline
    return self._buffer.readline(size)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Applications\scoop\apps\python\current\Lib\_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Applications\scoop\apps\python\current\Lib\gzip.py", line 499, in read
    if not self._read_gzip_header():
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Applications\scoop\apps\python\current\Lib\gzip.py", line 468, in _read_gzip_header
    last_mtime = _read_gzip_header(self._fp)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Applications\scoop\apps\python\current\Lib\gzip.py", line 423, in _read_gzip_header
    magic = fp.read(2)
            ^^^^^^^^^^
  File "D:\Applications\scoop\apps\python\current\Lib\gzip.py", line 97, in read
    self.file.read(size-self._length+read)
  File "F:\project\python\external_ftp_pipeline\.venv\Lib\site-packages\fsspec\spec.py", line 1856, in read
    out = self.cache._fetch(self.loc, self.loc + length)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\project\python\external_ftp_pipeline\.venv\Lib\site-packages\fsspec\caching.py", line 189, in _fetch
    self.cache = self.fetcher(start, end)  # new block replaces old
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\project\python\external_ftp_pipeline\.venv\Lib\site-packages\fsspec\implementations\ftp.py", line 333, in _fetch_range
    self.fs.ftp.getmultiline()
  File "D:\Applications\scoop\apps\python\current\Lib\ftplib.py", line 230, in getmultiline
    line = self.getline()
           ^^^^^^^^^^^^^^
  File "D:\Applications\scoop\apps\python\current\Lib\ftplib.py", line 212, in getline
    line = self.file.readline(self.maxline + 1)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Applications\scoop\apps\python\current\Lib\socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: timed out

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec/implementations/ftp.py, especially FTPFile._fetch_range and the FTPFileSystem setup shown in the issue. Reproduce the read against a FileZilla server and inspect the ABOR handling alongside the debug log. Done means the gzip file can be opened and read without the post-transfer timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.