tensorflow / tensorflow/datasets
`download=False` tries to retrieve token from GCE
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Short description
I expect would like this to work well offline, without any warnings or attempts to make http(s) requests to Google (or elsewhere).
2022-05-28 16:42:11.597299: W tensorflow/core/platform/cloud/google_auth_provider.cc:184] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "INVALID_ARGUMENT: Error executing an HTTP request: HTTP response code 400 with body '{
"error": "invalid_grant",
"error_description": "Bad Request"
}'". Retrieving token from GCE failed with "FAILED_PRECONDITION: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Could not resolve host: metadata"
Environment information
-
Operating System: macOS 12.4 (21F79)
-
Python version: 3.9.13
-
tensorflow-datasets/tfds-nightlyversion: 4.5.2 -
tensorflow/tf-nightlyversion: 2.9.1 -
Does the issue still exists with the last
tfds-nightlypackage (pip install --upgrade tfds-nightly) ?
Reproduction instructions
On a fresh virtualenv on a computer that has never used TensorFlow or TensorFlow datasets:
import new_dataset_based_on_voc_py
# Based ^ on https://github.com/tensorflow/datasets/blob/fdad1d9/tensorflow_datasets/object_detection/voc.py
import tensorflow_datasets as tfds
print("Loading:", new_dataset_based_on_voc_py._NewDataSetBasedOnVoc_CITATION)
tfds.load("new_dataset_based_on_voc_py",
data_dir=new_dataset_based_on_voc_py_data_dir,
download=False)
If you share a colab, make sure to update the permissions to share it.
Link to logs
2022-05-28 16:42:07 - tensorflow - DEBUG - Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
2022-05-28 16:42:07 - h5py._conv - DEBUG - Creating converter from 7 to 5
2022-05-28 16:42:07 - h5py._conv - DEBUG - Creating converter from 5 to 7
2022-05-28 16:42:07 - h5py._conv - DEBUG - Creating converter from 7 to 5
2022-05-28 16:42:07 - h5py._conv - DEBUG - Creating converter from 5 to 7
Loading: Geophysics dataset
2022-05-28 16:42:10 - absl - INFO - No config specified, defaulting to first: geo_phys/2022
2022-05-28 16:42:11 - absl - INFO - Constructing tf.data.Dataset new_dataset_based_on_voc_py for split None, from /tmp/<dir>/1.0.0
2022-05-28 16:42:11.597299: W tensorflow/core/platform/cloud/google_auth_provider.cc:184] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "INVALID_ARGUMENT: Error executing an HTTP request: HTTP response code 400 with body '{
"error": "invalid_grant",
"error_description": "Bad Request"
}'". Retrieving token from GCE failed with "FAILED_PRECONDITION: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Could not resolve host: metadata".
Expected behavior
No attempt at HTTP(S) requests if download=False. Maybe there's another config option I need to enable for this? - Otherwise it's a bug.
Additional context
N/A
EDIT: I found the try_download_gcs=True parameter to tfds.download.DownloadConfig… but when download=False shouldn't this also be False?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the tfds.load call and the tfds.download.DownloadConfig option try_download_gcs mentioned in the issue. Trace how download=False is handled and whether it reaches Google authentication or GCE metadata access. Done means offline loading makes no HTTP(S) or token-retrieval attempts, with a regression test covering the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100