huggingface / huggingface/datatrove
Slurm Cluster Size for FineWeb
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
I am working on a pipeline similar to FineWeb, and the time has come to start scaling up.
I am curious: what were the specs for the Slurm cluster used to produce FineWeb, and how long did the run take? It would save me a lot of trial and error in the provisioning process.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no file, test, or entry point. First confirm the FineWeb Slurm cluster specifications and runtime with maintainers, then document those details in the relevant project documentation once a location is identified.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, infrastructure
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100