fsspec / fsspec/filesystem_spec
`HTTPFileSystem` reads an incorrect amount of data
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Here's a minimal-ish repro:
from fsspec.implementations.http import HTTPFileSystem
remote_path = "https://huggingface.co/api/datasets/abisee/cnn_dailymail/parquet/3.0.0/train/0.parquet"
expected_data_size = 256540614
filesystem = HTTPFileSystem()
with filesystem.open(remote_path) as file:
total_read = 0
while data := file.read(256 * 1024):
total_read += len(data)
assert (
total_read == expected_data_size
), f"Data mismatch: {total_read} != {expected_data_size}"
# AssertionError: Data mismatch: 5767168 != 256540614
This issue causes Ray Data's from_huggingface API to break. See https://github.com/ray-project/ray/issues/54101
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the HTTPFileSystem entry point and run the minimal reproduction against the Hugging Face parquet URL. Trace how reads are bounded and compare the total bytes with the expected 256540614; done means the full file is read and Ray Data's from_huggingface API no longer breaks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100