NVIDIA-NeMo / NVIDIA-NeMo/Curator
Download and Extract - Should it support downloading to cloud instead of download to local only?
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
Currently Document Iteration (e.g. CommonCrawl) reads a file from local file system. We should allow it to use fsspec and support cloud storage too.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Document Iteration download and extraction path used by CommonCrawl, focusing on how it currently reads from the local filesystem. Review how fsspec could support cloud storage while preserving local behavior, then validate the behavior for both local and cloud-backed files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100