alibaba / alibaba/x-deeplearning
Checkpoint size问题
- Dominant language
- PureBasic
- Stars
- 4.3k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
我设置了个以embedding table为主的模型,设置ps_memory_m=64G,并且调整模型大小,使的placement正好占满memory(
再大一点就会报cannot placement的错误)。然后使用saver take了checkpoint,但是发现每个server的checkpoint大小都只有20G左右。请问为什么checkpoint的大小相比ps_memory_m要小这么多?XDL里什么部分使用了大量的内存,但是却不需要写到checkpoint里?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the saver checkpoint path and the memory accounting behind ps_memory_m and placement for the embedding-table model. Compare the memory reported for each server with the data actually serialized in its checkpoint; done means explaining the difference and identifying which memory is excluded from checkpoint output.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100