fsspec / fsspec/filesystem_spec
ssh filesystem and put_file are incompatible
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Using a ssh filesystem and put_files gives an error
import fsspec
import sshfs
fs = fsspec.filesystem("ssh", host='xx.xx.xx.xx', username='eouser')
fs.put_file(lpath="t", rpath="/home/me/t")
I receive an exception:
File "/home/me/miniconda3/envs/olci/lib/python3.9/site-packages/fsspec/callbacks.py", line 66, in relative_update
self.value += inc
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'
The reason is because the 'write' method in paramiko does not return the length of written data.
https://github.com/jbouse-debian/paramiko/blob/master/paramiko/file.py#L385
Unless paramiko changes their function, the method 'put_file' in spec.py should test segment_len to be None and skip the update... or use len(data) as a default as follows:
segment_len = f2.write(data)
if segment_len is None:
segment_len = len(data)
callback.relative_update(segment_len)
I'm not sure if this is a good solution...
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in spec.py at put_file, then inspect fsspec/callbacks.py and the Paramiko file.py write behavior linked in the report. Reproduce the SSH transfer with the provided example and verify that a write returning None no longer causes the callback TypeError while the transfer completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100