meta-pytorch / meta-pytorch/data
running FSSpecFileLister in ikernel doesn't work
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
Hi
This bug is following the conversation on discuss.pytorch.org
When running the following code in a jupyter kernel - the fs.protocol is not consistent
To reproduce - there is a need to update the url_to_fs call in /torchdata/datapipes/iter/load/fsspec.py file
fs, path = fsspec.core.url_to_fs(self.root, token='/Path/to/creds/credentials.json')
then run the following code
from torchdata.datapipes.iter import FSSpecFileLister
image_bucket = "gs://path/to/folder"
datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
list(file_dp)
in the second time running this code without restarting the kernel the URI returns without the gs:// but with the full path of the environment.
Versions
Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 12.3.1 (x86_64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.3)
CMake version: Could not collect
Libc version: N/A
Python version: 3.9.13 (main, May 24 2022, 21:28:31) [Clang 13.1.6 (clang-1316.0.21.2)] (64-bit runtime)
Python platform: macOS-12.3.1-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] facenet-pytorch==2.5.2
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.4
[pip3] pytorch-ignite==0.4.9
[pip3] torch==1.11.0
[pip3] torchdata==0.3.0
[pip3] torchvision==0.12.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in /torchdata/datapipes/iter/load/fsspec.py, focusing on the url_to_fs call used by FSSpecFileLister. Reproduce the example in a Jupyter kernel twice with the gs:// path and credentials token, then compare the returned URI and filesystem protocol. Done means repeated execution remains consistent without restarting the kernel.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100