tensorflow / tensorflow/datasets

Support for TFRecords preparation in ImageFolder

Open
#2,431 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Is your feature request related to a problem? Please describe.
tfds.ImageFolder replaces the legacy dataset named 'image_label_folder' in favor of a clean API that can be used to easily load datasets from local (or even GCS bucket) directories.

The download_and_prepare method:
https://github.com/tensorflow/datasets/blob/adb320ff04b6e93c561dacb2b647c8fcbfea92f3/tensorflow_datasets/core/folder_dataset/image_folder.py#L106-L112
present in ImageFolder doesn't provide the (undocumented) functionality of TFRecords preparation from custom directories unlike the deprecated 'image_label_folder', which used to well support that! It indeed encourages the users to directly load pipelines from the same directory structure without any overhead (very useful for smaller datasets) but the same might not be the desired option for medium-large datasets.

Describe the solution you'd like
Is there a clean (public) API available in TFDS for such easy TFRecords preparation from custom directories? I'd assume that interoperability of that with ImageFolder would greatly benefit users.

Describe alternatives you've considered
Are there any plans to support such a feature in the future? (if possible, we could implement the download_and_prepare method for ImageFolder referring to the implementation in older 'image_label_folder')

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tensorflow_datasets/core/folder_dataset/image_folder.py around download_and_prepare at lines 106-112, then compare the deprecated image_label_folder implementation for its TFRecords preparation behavior. Determine the intended public API and add coverage for preparing TFRecords from custom directories once the scope is agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data-engineering, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.