bigscience-workshop / bigscience-workshop/bigscience
mC4 sampling & pre-processing
- Lingua principale
- Shell
- Stelle
- 1k
- Fork
- 102
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
Hi @TevenLeScao,
I think there are some confusing and broken link in the [mC4 data preprocessing](https://github.com/bigscience-workshop/bigscience/tree/master/data/mc4) section. Can you take a look?
Both of the links are broken here,
1. [mc4_preprocessing](https://console.cloud.google.com/storage/browser/bigscience/mc4_preprocessing?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22)))
2. [mc4_sampled_raw](https://console.cloud.google.com/storage/browser/bigscience/mc4_sampled_raw?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22)))
The original link should be,
1. [mc4_preprocessing](https://console.cloud.google.com/storage/browser/bigscience/preprocessed_data/mc4_preprocessing;tab=objects?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false)
2. [mc4_sampled_raw](https://console.cloud.google.com/storage/browser/bigscience/preprocessed_data/mc4_sampled_raw?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false)
In addition to that, the multinomial data processing code to create the different language splits are in this pull request, https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/9
Here's few things,
1. Did you use this data for any one of your experiments?
2. If not then I think you can update the doc, https://github.com/bigscience-workshop/bigscience/tree/master/data/mc4
For reference purpose, if you want to keep the code, I'm happy to open a pull request here. If not I'll close the pull request from [bigscience/Megatron-Deepspeed](https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/9) repo.
Let me know what do you think.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia dalla documentazione data/mc4 collegata nell’issue e confronta i due link di storage di mC4 con le destinazioni previste. Verifica che le posizioni corrette siano ancora valide e chiarisci o aggiorna i riferimenti al preprocessing; il lavoro è concluso quando i link della documentazione funzionano e l’ambito relativo al codice di elaborazione interessato è stato deciso.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Ambito
- documentation
- Tipo di issue
- Documentazione
- Difficoltà
- 2/5
- Tempo stimato
- 1-3 ore
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100