tensorflow / tensorflow/datasets

[GSoC] Better fake data compression script

Open
#1,666 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Context: Since #1661, we have a script replace_fake_images.py which compress all images inside our fake data directory by replacing the random noise images by a uniform color image of the same shape/dtype.

Goal: However, when applied to our fake directory, some files seems to increase size, instead of being smaller (For the list of modified files, see #1634).
For example: imagenet2012/ILSVRC2012_img_train.tar increase by +50KB.
Some JPEG images are also badly compressed by the script (e.g. diabetic_retinopathy_detection/sample/1_left.jpeg).

The goal of this feature request would be:

Additionally, other improvement could be done in parallel by different persons:

  • Update the script to also compress audio files
  • Update the script to also compress video files.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tensorflow_datasets/scripts/replace_fake_images.py and inspect the fake files under tensorflow_datasets/testing/test_data/fake_examples. Run the script against representative archives and JPEGs, comparing input and output sizes while reviewing the PIL compression options. Done means the identified regressions are investigated and image or archive compression is improved; audio and video support are separate possible extensions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.