awslabs / awslabs/sagemaker-debugger

Error while running sagemaker-debugger with custom pytorch container and custom model

Open
#476 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
165
Forks
82
PR merge metrics
No merged PRs in 30d

Description

Hi,

I am running into the error below while running sagemaker-debugger with a custom pytorch container and custom model without sagemaker training. I added hooks to my model and loss using the below statements and tried running my training code but I am running into this error:

```
FileNotFoundError: [Errno 2] No such file or directory: 'smdebug_outputs/collections/000000000/worker_0_collections.json.tmp'
```
where 'smdebug_outputs' is the output directory given.

I inserted the following snippet in my code for inserting hooks:
```
import smdebug.pytorch as smd
hook = smd.Hook(out_dir)
hook.register_module(net)

# Inside training loop
loss = net(inputs)
hook.record_tensor_value(tensor_name="loss", tensor_value=loss)
```

Is there some other modifications needed to get sagemaker-debugger running on a custom model and container?

Contributor guide

Open the contributing guide

Research direction

Start with the custom-container training code and the smdebug.pytorch Hook setup shown in the report, then reproduce the FileNotFoundError using the supplied output directory. Inspect the generated smdebug_outputs/collections path during the run and verify the required setup for a custom model without SageMaker training. Done means the run completes and records the loss without the missing-file error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
devtools, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.