tensorflow / tensorflow/cloud

How to load a dataset from google cloud storage with tensorflow cloud?

Open
#136 2 comments 0 reactions 1 assignee View on GitHub

@pavithrasv is already working on this.

Since Aug 20, 2020.

bug
Dominant language
Python
Stars
383
Forks
93
Avg merge
1d 3h
Merged PRs (30d)
1

Description

Tensorflow cloud configuration:

  GCP_BUCKET = "stereo-train"

  tfc.run(
    requirements_txt="requirements.txt",
    chief_config=tfc.MachineConfig(
      cpu_cores=8,
      memory=30,
      accelerator_type=tfc.AcceleratorType.NVIDIA_TESLA_T4,
      accelerator_count=1,
    ),
    docker_image_bucket_name=GCP_BUCKET,
  )

And I have a bucket called gs://stereo-train that contains the dataset. The exact location of the dataset is:

gs://stereo-train/data_scene_flow/training/dat

However, when using this location like so:

tf.keras.preprocessing.image_dataset_from_directory("gs://stereo-train/data_scene_flow/training/dat", image_size=(375,1242),\
                                                         batch_size=6, shuffle=False, label_mode=None)

Behavior:

getting the error that "gs://stereo-train/data_scene_flow/training/dat" doesn't exist

Expected behavior:
tf.keras.preprocessing.image_dataset_from_directory should know that there's a gs bucket associated with the account and the dataset should be loaded.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.