fsspec / fsspec/filesystem_spec
FTPFileSystem open file cannot read from filezilla server
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
The ABOR command is sent when the ftp file download is finished, the server no longer responds and raises the TimeoutError
code:
def test1():
fs: FTPFileSystem = fsspec.filesystem('ftp', asynchronous=True, host='127.0.0.1', port=21, username='admin',
password='admin', timeout=10)
with fs.open(fs.ls('/realtime_update')[0]['name'], compression='gzip') as f:
f.read()
fsspec.implementations.ftp.FTPFile._fetch_range
# stop transfer, we got enough bytes for this block
self.fs.ftp.abort()
debug log
*get* '150 Starting data transfer.\n'
*resp* '150 Starting data transfer.'
*put urgent* b'ABOR\r\n'
*get* '226 Operation successful\n'
Traceback (most recent call last):
File "F:\project\python\external_ftp_pipeline\.venv\Lib\site-packages\fsspec\implementations\ftp.py", line 323, in _fetch_range
self.fs.ftp.retrbinary(
File "D:\Applications\scoop\apps\python\current\Lib\ftplib.py", line 441, in retrbinary
callback(data)
File "F:\project\python\external_ftp_pipeline\.venv\Lib\site-packages\fsspec\implementations\ftp.py", line 315, in callback
raise TransferDone
fsspec.implementations.ftp.TransferDone
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "F:\project\python\external_ftp_pipeline\src\external_ftp\gongshang_ftp.py", line 154, in <module>
asyncio.run(main())
File "D:\Applications\scoop\apps\python\current\Lib\asyncio\runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "D:\Applications\scoop\apps\python\current\Lib\asyncio\runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Applications\scoop\apps\python\current\Lib\asyncio\base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "F:\project\python\external_ftp_pipeline\src\external_ftp\gongshang_ftp.py", line 136, in main
for tag, data in load_ftp_infos():
File "F:\project\python\external_ftp_pipeline\src\external_ftp\gongshang_ftp.py", line 127, in load_ftp_infos
for data in decode_lines(f):
File "F:\project\python\external_ftp_pipeline\src\external_ftp\gongshang_ftp.py", line 117, in decode_lines
for line in source:
File "D:\Applications\scoop\apps\python\current\Lib\gzip.py", line 399, in readline
return self._buffer.readline(size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Applications\scoop\apps\python\current\Lib\_compression.py", line 68, in readinto
data = self.read(len(byte_view))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Applications\scoop\apps\python\current\Lib\gzip.py", line 499, in read
if not self._read_gzip_header():
^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Applications\scoop\apps\python\current\Lib\gzip.py", line 468, in _read_gzip_header
last_mtime = _read_gzip_header(self._fp)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Applications\scoop\apps\python\current\Lib\gzip.py", line 423, in _read_gzip_header
magic = fp.read(2)
^^^^^^^^^^
File "D:\Applications\scoop\apps\python\current\Lib\gzip.py", line 97, in read
self.file.read(size-self._length+read)
File "F:\project\python\external_ftp_pipeline\.venv\Lib\site-packages\fsspec\spec.py", line 1856, in read
out = self.cache._fetch(self.loc, self.loc + length)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\project\python\external_ftp_pipeline\.venv\Lib\site-packages\fsspec\caching.py", line 189, in _fetch
self.cache = self.fetcher(start, end) # new block replaces old
^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\project\python\external_ftp_pipeline\.venv\Lib\site-packages\fsspec\implementations\ftp.py", line 333, in _fetch_range
self.fs.ftp.getmultiline()
File "D:\Applications\scoop\apps\python\current\Lib\ftplib.py", line 230, in getmultiline
line = self.getline()
^^^^^^^^^^^^^^
File "D:\Applications\scoop\apps\python\current\Lib\ftplib.py", line 212, in getline
line = self.file.readline(self.maxline + 1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Applications\scoop\apps\python\current\Lib\socket.py", line 706, in readinto
return self._sock.recv_into(b)
^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: timed out
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fsspec/implementations/ftp.py, especially FTPFile._fetch_range and the FTPFileSystem setup shown in the issue. Reproduce the read against a FileZilla server and inspect the ABOR handling alongside the debug log. Done means the gzip file can be opened and read without the post-transfer timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100