huggingface / huggingface/datatrove
How to look into the processed data?
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Hi,
After running `tokenize_from_hf_to_s3.py`, I would like to inspect the resulting data. But I find that the current data is in a binary file (`.ds`). is there a way to allow me to look into the data?
Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with tokenize_from_hf_to_s3.py and inspect how it produces the binary .ds output. Determine a reproducible way to inspect the resulting data, and consider the work complete when users can follow documented steps to view it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100