tensorflow / tensorflow/datasets

Add recursive support for `GPath`

Open
#2,634 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

contributions welcome enhancement
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Currently, tf.io.gfile.glob does not support recursive: https://github.com/tensorflow/tensorflow/issues/39565

Example to demonstrate the issue:

import glob
import tensorflow as tf

tf.io.gfile.makedirs('/tmp/a/b/c')

tf.io.gfile.glob('/tmp/a/**')  # ['/tmp/a/b']
glob.glob('/tmp/a/**', recursive=True)  # ['/tmp/a/', '/tmp/a/b', '/tmp/a/b/c']

This make GPath().glob('**/x.txt') inconsistent with python glob.

It would be nice to add recursive support in our GPath.glob and GPath.rglob wrapper around tf.io.gfile. This would be done by a combinaison of .iterdir(), .isdir() (to recurse) and fnmatch.fnmatch() to check that the file match the pattern.

Implementation should go in core/utils/gpath.py, tests in core/utils/gpath_test.py

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in core/utils/gpath.py and read the existing tests in core/utils/gpath_test.py. Check how GPath.glob and GPath.rglob currently use tf.io.gfile, then run the GPath tests while covering recursive patterns such as **/x.txt. Done means recursive globbing matches the stated Python glob behavior, including nested directories.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.