fsspec / fsspec/filesystem_spec
http get() call creates malformed dest path
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
See original discussion here and a linked bug here
Issue
Creating an http file system and using an rpath with query params results in a malformed destination path.
Steps to Reproduce
Using the test below, update lpath to a path you can write to.
def test_http_output():
kwargs = {}
fs = fsspec.implementations.http.HTTPFileSystem(fsspec.filesystem("https", **kwargs))
expected_output_path = "/var/myloc/outputfile.txt"
rpath = "https://httpbin.org/gzip?test=value"
lpath = expected_output_path
fs.get(rpath, lpath)
This behavior only happens (in my testing) when the rpath uses query params. That triggers exists to be set to True because has_magic returns True.
It also requires lpath and rpath to be strings rather than lists.
Expected Results
A file is written to /var/myloc/outputfile.txt
Actual Results
A file is written to /var/myloc/outputfile.txt/gzip\?test=value
Notes
It's also possible to test the underlying path construction using testutils.py test_other_paths and adding the following parameterized test: (["/path1"], "/path2/path3.txt", True, ["/path2/path3.txt"])
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with HTTPFileSystem.get and the path-construction logic exercised by the reproduction, then inspect testutils.py and its test_other_paths cases. Reproduce the query-parameter case with string paths and add or update the parameterized test; done means the destination is /var/myloc/outputfile.txt rather than a path containing the remote filename and query string.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100