huggingface / huggingface/datatrove
I would like to get help from Datatrove enthusiasts regarding issues I'm facing while running the example script.
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Hello, Datatrove enthusiasts,
Nice to meet you all.
Recently, I've been working on the Datatrove library and I'm trying to run a sample script, `process_common_crawl_dump.py` from the following link: [Datatrove GitHub](https://github.com/huggingface/datatrove/blob/main/examples/process_common_crawl_dump.py).
I've made a couple of changes to the script: I've reduced the number of tasks from 8000 to 4 and updated `randomize_start_duration` to `randomize_start`. However, after running the script, I encountered some issues.
Here is the accounting history that I received:

Additionally, I believe these logs are stored on my S3:



I was expecting to get an output as a result, but there is no any output directories or files.
only I got logs files

For reference, here is my `slurm.conf` file:

I've tried running the script multiple times, but I always get the same result. I'm not sure if this is the right place to ask for help, but I would appreciate any assistance from fellow Datatrove lovers.
Thank you!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.