NVIDIA-NeMo / NVIDIA-NeMo/Curator

[FEA] Cloud-compatibility tests for Text Components (use fsspec, not os)

Open
#984 2 comments 1 reaction 1 assignee View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1.8k
Forks
328
Avg merge
4d 5h
Merged PRs (30d)
30

Description

Cloud-compatibility tests for Text Components (use fsspec, not os)

Goal

Ensure all Text Components use fsspec for paths/IO. Ban os.path, pathlib.Path, glob, shutil for URI work.

Why

os/pathlib/glob assume local FS and break on s3://, gs://, abfs://, http(s)://.

What breaks → What to use

Exists/stat: os.path.exists("s3://…") → False
➜ fs.exists(p), fs.info(p)

Open/write: open("s3://…") fails
➜ with fs.open(p, "wb") as f: …

Join/normalize: os.path.normpath("s3://b/a/../c") mangles protocol
➜ posixpath.join(root, "a", "b") (after url_to_fs)

List/walk: os.listdir, os.walk fail
➜ fs.ls(root), fs.walk(root)

Glob: glob.glob("s3://…") → []
➜ fs.glob("s3://bucket/**/*.json")

Mk/rm/mv/cp: os.makedirs, shutil.rmtree/move
➜ fs.makedirs, fs.rm, fs.mv, fs.cp

What works:
  • os.path.join → Allow this as this works (and we dont care about windows support)
Refactors stuff like:
  • open(uri, mode) → fs.open(p, mode)
  • os.listdir/os.walk → fs.ls/fs.walk
  • glob.glob → fs.glob
  • os.makedirs/os.remove/shutil.* → fs.makedirs/fs.rm/fs.mv/fs.cp

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.