awslabs / awslabs/sagemaker-debugger
Compatibility with gradient accumulation
- Dominant language
- Python
- Stars
- 165
- Forks
- 82
- PR merge metrics
- No merged PRs in 30d
Description
I'm bringing my own PyTorch training script, and I'm interested in using SM Debugger to profile function calls in my training jobs. The [API Glossary](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/api.md#glossary) states:
> Step: Step means one the work done by the training job for one batch (i.e. forward and backward pass).
I assume I will have to register my module with `hook.register_module(module)` in the training script for SM Debugger to work at all. I further assume that SM Debugger then registers its own hooks into the module's forward() and/or backward() passes to track when a "step" happens.
However, my training script accumulates gradients from several forward() passes before running a single backward() pass.
My questions:
1. Will this interfere with the functionality of SM Debugger?
2. Assuming this is okay, does SM Debugger consider the forward() or the backward() pass to be one "step"?
Contributor guide
Research direction
Start with docs/api.md#glossary and the hook.register_module entry point mentioned in the issue. Check how SM Debugger observes forward and backward passes during gradient accumulation, then document whether accumulation affects profiling and which operation represents a step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, pytorch
- Domain
- devtools, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100