NVIDIA-NeMo / NVIDIA-NeMo/Curator
[FEA] Cloud-compatibility tests for Text Components (use fsspec, not os)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
Cloud-compatibility tests for Text Components (use fsspec, not os)
Goal
Ensure all Text Components use fsspec for paths/IO. Ban os.path, pathlib.Path, glob, shutil for URI work.
Why
os/pathlib/glob assume local FS and break on s3://, gs://, abfs://, http(s)://.
What breaks → What to use
Exists/stat: os.path.exists("s3://…") → False
➜ fs.exists(p), fs.info(p)
Open/write: open("s3://…") fails
➜ with fs.open(p, "wb") as f: …
Join/normalize: os.path.normpath("s3://b/a/../c") mangles protocol
➜ posixpath.join(root, "a", "b") (after url_to_fs)
List/walk: os.listdir, os.walk fail
➜ fs.ls(root), fs.walk(root)
Glob: glob.glob("s3://…") → []
➜ fs.glob("s3://bucket/**/*.json")
Mk/rm/mv/cp: os.makedirs, shutil.rmtree/move
➜ fs.makedirs, fs.rm, fs.mv, fs.cp
What works:
- os.path.join → Allow this as this works (and we dont care about windows support)
Refactors stuff like:
- open(uri, mode) → fs.open(p, mode)
- os.listdir/os.walk → fs.ls/fs.walk
- glob.glob → fs.glob
- os.makedirs/os.remove/shutil.* → fs.makedirs/fs.rm/fs.mv/fs.cp
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.