NVIDIA-NeMo / NVIDIA-NeMo/Curator

Cloud Storage Compatibility (fsspec) Throughout

Open
#1,709 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1.8k
Forks
328
Avg merge
4d 5h
Merged PRs (30d)
30

Description

Replace all os.path, pathlib.Path, glob, and shutil I/O in processing stages with fsspec-backed equivalents so S3, GCS, and Azure paths work identically to local paths.

Why: All production data lives in object storage. Today, Curator's download stages support cloud I/O but the processing stages use local-only primitives, breaking on s3:// or gs:// URIs. This forces users to sync data locally before processing — untenable at large scale.
Definition of Done:

  • All file reads/writes in stages/ use fsspec.open() or equivalent
  • s3://, gs://, az://, hf://, and file:// paths work in all stages without code changes
  • FilePartitioningStage resolves cloud paths correctly (issue #1568)
  • CI tests added for S3-compatible storage (e.g., MinIO in CI)
  • No dependency on os.path, pathlib.Path, glob, or shutil for URI handling in stage code (enforced by linter rule per issue #984)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inventorying URI handling throughout stages/ and read FilePartitioningStage alongside issue #1568. Then inspect the CI setup for adding MinIO coverage. Done means all listed URI schemes work across stages, local-only path primitives are absent from stage URI handling, and the linter requirement is enforced.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, azure, gcp, python
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.