pytorch / pytorch/pytorch

Improve documentation of assumptions for symmetric memory

Open
#173,514 1 comment 0 reactions 0 assignees View on GitHub
module: docs module: symm_mem triaged
Dominant language
Python
Stars
103k
Forks
29.6k
PR merge metrics
PR metrics pending

Description

### 📚 The doc issue

Current docs for [symmetric memory ](https://docs.pytorch.org/docs/stable/symmetric_memory.html) specify that "The empty and rendezvous functions must be called in the same order on all ranks in the group." but IMO, this should be improved:

1. This sentence appears under the “Hello World” example section, which isn't ideal. It would be good to repeat it within the API reference or provide a link to a specific section outlining all assumptions
2. Is my understanding correct that symmetric memory also expects all ranks to provide the same `size` for `empty` ? If this is the case, it should be documented somewhere. If not, how would the strided memory look like / interact with symm_mem ops ?
3. I believe the same assumptions apply for the case where a [memory pool](https://docs.pytorch.org/docs/stable/symmetric_memory.html#using-memory-pool) is used : it would be good to add this to that section as well (maybe through a link).

While looking for this, I noticed that https://github.com/pytorch/pytorch/issues/164281 is still open but I believe it can be closed.

CC: @kwen2501

### Suggest a potential alternative/fix

_No response_

cc @svekars @sekyondaMeta @AlannaBurke

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.