tensorflow / tensorflow/datasets
Add script to export tensorflow_data_validation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Currently, there is an option to generate TFDV statistics for FACET during download_and_prepare. However, it would be best to separate those two steps.
builder = tfds.builder('dataset')
builder.download_and_prepare()
generate_statistics(builder, dst=builder.data_dir)
Then we should add a new script in scripts/generate_statistics.py which generate statistics for one dataset and export them to gs://. We would automatically generate statistics for all datasets.
When user pre-load the DatasetInfo, it would also fetch the statistics.json from GCS, so user could visualize FACET for all default TFDS datasets with the existing tfds.show_statistics.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing TFDV statistics generation in download_and_prepare and the generate_statistics entry point. Review scripts/generate_statistics.py, DatasetInfo loading, and tfds.show_statistics to define how one dataset exports statistics to gs:// and how preloading fetches statistics.json. Done means statistics generation is separate, all datasets can be processed automatically, and FACETS visualization works through tfds.show_statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- cloud, data, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100