bigscience-workshop / bigscience-workshop/data-preparation
Welcome to try SailCraft - A data cleaning tool built upon this repository
Open
- Dominant language
- Jupyter Notebook
- Stars
- 318
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
We extend our gratitude to the authors of this repository! Your documentation and code have greatly benefited the community.
We have used this repo in building the data processing pipeline tool [SailCraft](https://github.com/sail-sg/sailcraft).
It consists of four stages: initial data cleaning, near deduplication, exact deduplication, and a second round of data cleaning.
Many thanks for your contribution for open research! And we welcome the developers to try SailCraft.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.