meta-pytorch / meta-pytorch/data

running FSSpecFileLister in ikernel doesn't work

Open
#497 21 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

🐛 Describe the bug

Hi
This bug is following the conversation on discuss.pytorch.org
When running the following code in a jupyter kernel - the fs.protocol is not consistent

To reproduce - there is a need to update the url_to_fs call in /torchdata/datapipes/iter/load/fsspec.py file

   fs, path = fsspec.core.url_to_fs(self.root, token='/Path/to/creds/credentials.json')

then run the following code

from torchdata.datapipes.iter import FSSpecFileLister
image_bucket = "gs://path/to/folder"
datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
list(file_dp)

in the second time running this code without restarting the kernel the URI returns without the gs:// but with the full path of the environment.

Versions

Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.3.1 (x86_64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.3)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.13 (main, May 24 2022, 21:28:31) [Clang 13.1.6 (clang-1316.0.21.2)] (64-bit runtime)
Python platform: macOS-12.3.1-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] facenet-pytorch==2.5.2
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.4
[pip3] pytorch-ignite==0.4.9
[pip3] torch==1.11.0
[pip3] torchdata==0.3.0
[pip3] torchvision==0.12.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in /torchdata/datapipes/iter/load/fsspec.py, focusing on the url_to_fs call used by FSSpecFileLister. Reproduce the example in a Jupyter kernel twice with the gs:// path and credentials token, then compare the returned URI and filesystem protocol. Done means repeated execution remains consistent without restarting the kernel.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.