huggingface / huggingface/datatrove

I would like to get help from Datatrove enthusiasts regarding issues I'm facing while running the example script.

Open
#235 31 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

Hello, Datatrove enthusiasts,

Nice to meet you all.

Recently, I've been working on the Datatrove library and I'm trying to run a sample script, `process_common_crawl_dump.py` from the following link: [Datatrove GitHub](https://github.com/huggingface/datatrove/blob/main/examples/process_common_crawl_dump.py).

I've made a couple of changes to the script: I've reduced the number of tasks from 8000 to 4 and updated `randomize_start_duration` to `randomize_start`. However, after running the script, I encountered some issues.

Here is the accounting history that I received:
![Accounting History](https://github.com/huggingface/datatrove/assets/148059188/9d7f660c-1540-43f4-b1e8-594f2db3443e)

Additionally, I believe these logs are stored on my S3:
![S3 Log 1](https://github.com/huggingface/datatrove/assets/148059188/d359f8f0-8cc8-4964-9840-d300a786dad0)
![image](https://github.com/huggingface/datatrove/assets/148059188/c232ffc4-20c0-4c1a-844f-26569ef1ec3d)
![image](https://github.com/huggingface/datatrove/assets/148059188/69fb988d-e288-4a50-8b88-b6dfe8caa793)
I was expecting to get an output as a result, but there is no any output directories or files.
only I got logs files
![Expected Output](https://github.com/huggingface/datatrove/assets/148059188/35b5061a-2891-4ec2-999f-fb51ef2c151c)

For reference, here is my `slurm.conf` file:
![Slurm.conf](https://github.com/huggingface/datatrove/assets/148059188/a003d0d1-dd23-4b9c-9c2f-a69dbf574016)

I've tried running the script multiple times, but I always get the same result. I'm not sure if this is the right place to ask for help, but I would appreciate any assistance from fellow Datatrove lovers.

Thank you!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.