aws / aws/amazon-sagemaker-examples
[AWS-Sagemekar] TensorBoardOutputConfig
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I'm using sagemaker (2.111.0) PyTorch estimator. I have configured the tensorboard log directory as follows:
**from sagemaker.debugger import TensorBoardOutputConfig
tensorboard_output_config = TensorBoardOutputConfig(
s3_output_path=f"s3://",
container_local_output_path='/opt/ml/checkpoints/tb'
)**
I'm getting the following error:
**botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: "LocalPath" of "TensorBoardOutputConfig" cannot start with the following reserved path: [/opt/ml, /tmp, /usr/local/nvidia]**
I have configured it like this before but it is not allowing me to use /opt/ml now. And if I don't do so the tensorboard logs won't be copied to my s3 bucket in real-time, rendering tensorboard useless.
The reason for having tensorboard logs in real-time is to save cost if my training job is going sideways. Now there is no way to know if my job is running as expected until the job is finished.
Any help is much appreciated!
Best regards,
Aditya
Contributor guide
Research direction
Start by reproducing the CreateTrainingJob failure with the provided PyTorch estimator and TensorBoardOutputConfig using sagemaker 2.111.0. Review the TensorBoardOutputConfig handling and the relevant SageMaker example entry point; done means establishing a supported configuration or a documented resolution for real-time TensorBoard log upload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, pytorch
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100