awslabs / awslabs/sagemaker-debugger

Compatibility with gradient accumulation

Open
#426 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
165
Forks
82
PR merge metrics
No merged PRs in 30d

Description

I'm bringing my own PyTorch training script, and I'm interested in using SM Debugger to profile function calls in my training jobs. The [API Glossary](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/api.md#glossary) states:

> Step: Step means one the work done by the training job for one batch (i.e. forward and backward pass).

I assume I will have to register my module with `hook.register_module(module)` in the training script for SM Debugger to work at all. I further assume that SM Debugger then registers its own hooks into the module's forward() and/or backward() passes to track when a "step" happens.

However, my training script accumulates gradients from several forward() passes before running a single backward() pass.

My questions:
1. Will this interfere with the functionality of SM Debugger?
2. Assuming this is okay, does SM Debugger consider the forward() or the backward() pass to be one "step"?

Contributor guide

Open the contributing guide

Research direction

Start with docs/api.md#glossary and the hook.register_module entry point mentioned in the issue. Check how SM Debugger observes forward and backward passes during gradient accumulation, then document whether accumulation affects profiling and which operation represents a step.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
devtools, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.