awslabs / awslabs/sagemaker-debugger
Error while running sagemaker-debugger with custom pytorch container and custom model
- Dominant language
- Python
- Stars
- 165
- Forks
- 82
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I am running into the error below while running sagemaker-debugger with a custom pytorch container and custom model without sagemaker training. I added hooks to my model and loss using the below statements and tried running my training code but I am running into this error:
```
FileNotFoundError: [Errno 2] No such file or directory: 'smdebug_outputs/collections/000000000/worker_0_collections.json.tmp'
```
where 'smdebug_outputs' is the output directory given.
I inserted the following snippet in my code for inserting hooks:
```
import smdebug.pytorch as smd
hook = smd.Hook(out_dir)
hook.register_module(net)
# Inside training loop
loss = net(inputs)
hook.record_tensor_value(tensor_name="loss", tensor_value=loss)
```
Is there some other modifications needed to get sagemaker-debugger running on a custom model and container?
Contributor guide
Research direction
Start with the custom-container training code and the smdebug.pytorch Hook setup shown in the report, then reproduce the FileNotFoundError using the supplied output directory. Inspect the generated smdebug_outputs/collections path during the run and verify the required setup for a custom model without SageMaker training. Done means the run completes and records the loss without the missing-file error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, pytorch
- Domain
- devtools, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100