fsspec / fsspec/filesystem_spec

Dealing with an FTP server that doesn't support offsets

Open
#1,116 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Running code like:

with fsspec.open(an_ftp_url) as fin:
    while True:
        buf = fin.read(16*1024)
        if not buf:
            break

raises a error_temp: 426 Connection closed; transfer aborted.
here:
https://github.com/fsspec/filesystem_spec/blob/62bb12e681b9e3dbd25df3991fd71552ea7654ee/fsspec/implementations/ftp.py#L314

The server doesn't support the rest argument.
From the docs: https://docs.python.org/3/library/ftplib.html#ftplib.FTP.transfercmd it sounds like this is not uncommon:

If the server does not recognize the REST command, an [error_reply](https://docs.python.org/3/library/ftplib.html#ftplib.error_reply) exception will be raised. If this happens, simply call [transfercmd()](https://docs.python.org/3/library/ftplib.html#ftplib.FTP.transfercmd) without a rest argument.

I'm curious if you have any suggestions on how do get an FTP server like this to work with fsspec?

The obvious answer of just reading the entire buffer into memory buf = fin.read() is not satisfying. I'd like to be able to copy the file in chunks to a destination and not load the entire thing into memory.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec/implementations/ftp.py around line 314 and trace the chunked read path through the FTP transfer command. The desired behavior is for servers that reject REST to support incremental reads and copying without loading the entire file into memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.