fsspec / fsspec/filesystem_spec

http get() call creates malformed dest path

Open
#1,505 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

See original discussion here and a linked bug here

Issue

Creating an http file system and using an rpath with query params results in a malformed destination path.

Steps to Reproduce

Using the test below, update lpath to a path you can write to.

def test_http_output():
    kwargs = {}
    fs = fsspec.implementations.http.HTTPFileSystem(fsspec.filesystem("https", **kwargs))
    expected_output_path = "/var/myloc/outputfile.txt"
    rpath = "https://httpbin.org/gzip?test=value"
    lpath = expected_output_path
   fs.get(rpath, lpath)

This behavior only happens (in my testing) when the rpath uses query params. That triggers exists to be set to True because has_magic returns True.

It also requires lpath and rpath to be strings rather than lists.

Expected Results

A file is written to /var/myloc/outputfile.txt

Actual Results

A file is written to /var/myloc/outputfile.txt/gzip\?test=value

Notes

It's also possible to test the underlying path construction using testutils.py test_other_paths and adding the following parameterized test: (["/path1"], "/path2/path3.txt", True, ["/path2/path3.txt"])

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with HTTPFileSystem.get and the path-construction logic exercised by the reproduction, then inspect testutils.py and its test_other_paths cases. Reproduce the query-parameter case with string paths and add or update the parameterized test; done means the destination is /var/myloc/outputfile.txt rather than a path containing the remote filename and query string.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.