tensorflow / tensorflow/datasets
Count duplicates… e.g., `tf.data.experimental.unique` + `tf.data.experimental.cardinality`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Is your feature request related to a problem? Please describe.
My NN results are good. Too good. So I'm trying to figure out why it's so good. Maybe there are duplicates somewhere?
Describe the solution you'd like
Would be great to just count the duplicates.
Describe alternatives you've considered
I'm sure I'll end up hashing everything matrix into strings or [more efficiently… assuming it fits] scalars. Then deduplicate that list.
Additional context
Might be good to have a proper summary, combining https://stackoverflow.com/a/60877708 to get the counts per label, with this solution (unique counts per label, counts per split, and total counts for each).
EDIT: Just found https://www.tensorflow.org/tfx/guide/tfdv, https://www.tensorflow.org/tfx/data_validation/get_started and reading https://pair-code.github.io/facets/ now…
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository file or test is identified. Start by reviewing the tf.data.experimental.unique and tf.data.experimental.cardinality APIs and the linked Stack Overflow and TensorFlow Data Validation references. Done would require an agreed approach for reporting duplicate counts per label, per split, and for the full dataset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100