aws / aws/amazon-sagemaker-examples

[AWS-Sagemekar] TensorBoardOutputConfig

Open
#3,620 7 comments 4 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I'm using sagemaker (2.111.0) PyTorch estimator. I have configured the tensorboard log directory as follows:

**from sagemaker.debugger import TensorBoardOutputConfig
tensorboard_output_config = TensorBoardOutputConfig(
s3_output_path=f"s3://",
container_local_output_path='/opt/ml/checkpoints/tb'
)**

I'm getting the following error:
**botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: "LocalPath" of "TensorBoardOutputConfig" cannot start with the following reserved path: [/opt/ml, /tmp, /usr/local/nvidia]**

I have configured it like this before but it is not allowing me to use /opt/ml now. And if I don't do so the tensorboard logs won't be copied to my s3 bucket in real-time, rendering tensorboard useless.

The reason for having tensorboard logs in real-time is to save cost if my training job is going sideways. Now there is no way to know if my job is running as expected until the job is finished.

Any help is much appreciated!

Best regards,
Aditya

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the CreateTrainingJob failure with the provided PyTorch estimator and TensorBoardOutputConfig using sagemaker 2.111.0. Review the TensorBoardOutputConfig handling and the relevant SageMaker example entry point; done means establishing a supported configuration or a documented resolution for real-time TensorBoard log upload.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.