NVIDIA-NeMo / NVIDIA-NeMo/Curator
Allow folks to write/provide a dataset_name and possibly naming scheme for output (jsonl, parquet etc)
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
Is your feature request related to a problem? Please describe.
We should explore how to enable folks to write/provide a dataset_name and possibly naming scheme for output (jsonl, parquet etc).
Ray data does this via FilenameProvider . https://docs.ray.io/en/latest/data/api/doc/ray.data.datasource.FilenameProvider.html#ray.data.datasource.FilenameProvider
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.