tensorflow / tensorflow/datasets
[GSoC] Better fake data compression script
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Context: Since #1661, we have a script replace_fake_images.py which compress all images inside our fake data directory by replacing the random noise images by a uniform color image of the same shape/dtype.
Goal: However, when applied to our fake directory, some files seems to increase size, instead of being smaller (For the list of modified files, see #1634).
For example: imagenet2012/ILSVRC2012_img_train.tar increase by +50KB.
Some JPEG images are also badly compressed by the script (e.g. diabetic_retinopathy_detection/sample/1_left.jpeg).
The goal of this feature request would be:
- Investigate why some archives/image size increase after being compressed.
- Find a way to better compress the images / archives (e.g. by playing with the compression parameters of PIL https://pillow.readthedocs.io/en/5.1.x/handbook/image-file-formats.html#jpeg)
Additionally, other improvement could be done in parallel by different persons:
- Update the script to also compress audio files
- Update the script to also compress video files.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tensorflow_datasets/scripts/replace_fake_images.py and inspect the fake files under tensorflow_datasets/testing/test_data/fake_examples. Run the script against representative archives and JPEGs, comparing input and output sizes while reviewing the PIL compression options. Done means the identified regressions are investigated and image or archive compression is improved; audio and video support are separate possible extensions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100