Question about distributed gradient calculation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 213
- PR merge metrics
- No merged PRs in 30d
Description
Dear author,
I'm wondering the difference between w/o gather_with_grad when collect tensor from different gpus:
https://github.com/LAION-AI/CLAP/blob/6b1b4b5b4b87f4e19d3836d2ae7d7272e1c69410/src/laion_clap/clap_module/loss.py#L59C11-L59C11
Looks like the final return (all_audio_features & all_text_features) is the same?
Another related question is, during distributed training, how is the grad calculated?
From my understanding of the code, for each gpu, it collects audio_feature & text_feature from all gpus and then do loss calculation and backward separately.
Does it mean duplicate calculation as the gathered features should be the same for each gpu?
Please correct my understanding if it is wrong, thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/laion_clap/clap_module/loss.py at the linked line and trace the distributed feature-gathering path. Compare the behavior with and without gather_with_grad, then follow the training loss and backward flow across GPUs. Done means the gradient behavior and any difference between the two paths are clearly explained or documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100