tensorflow / tensorflow/tensorboard

tensorboard not working with S3

Open
#4,255 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

core:backend stat:community support type:bug
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Environment information (required)

Mac 10.14.6
Tensorboard 2.0.0
botocore 1.19.1
boto3 1.16.0
tensorflow 2.0.0 mkl_py37hda344b4_0
tensorflow-base 2.0.0 mkl_py37h66b1bf0_0
tensorflow-estimator 2.0.0 pyh2649769_0

AWS credentials are available in the ~/.aws/credentails file in profile default

Issue description

AWS_LOG_LEVEL=1 AWS_DEFAULT_PROFILE=default S3_REGION=eu-west-1 tensorboard --logdir s3://gourav-bucket/gourav-data/tensorboard/

gives the following errors:

Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.0.0 at http://localhost:6006/ (Press CTRL+C to quit)
2020-10-24 23:13:51.056759: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
2020-10-24 23:13:51.056861: E tensorflow/core/platform/s3/aws_logging.cc:60] Http request to retrieve credentials failed
2020-10-24 23:13:51.056910: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 0 ms before attempting again.
2020-10-24 23:13:52.057796: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
2020-10-24 23:13:52.057882: E tensorflow/core/platform/s3/aws_logging.cc:60] Http request to retrieve credentials failed
2020-10-24 23:13:52.057923: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 2000 ms before attempting again.
2020-10-24 23:13:55.062484: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
2020-10-24 23:13:55.062540: E tensorflow/core/platform/s3/aws_logging.cc:60] Http request to retrieve credentials failed
2020-10-24 23:13:55.062570: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 4000 ms before attempting again.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported command with TensorBoard 2.0.0 and the listed AWS environment variables, then investigate the credential retrieval path indicated by the S3 error output. Done means TensorBoard can access the specified S3 log directory using the default profile without repeated credential-request failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.