alibaba / alibaba/x-deeplearning

Checkpoint size问题

Open
#346 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
PureBasic
Stars
4.3k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

我设置了个以embedding table为主的模型,设置ps_memory_m=64G,并且调整模型大小,使的placement正好占满memory(
再大一点就会报cannot placement的错误)。然后使用saver take了checkpoint,但是发现每个server的checkpoint大小都只有20G左右。请问为什么checkpoint的大小相比ps_memory_m要小这么多?XDL里什么部分使用了大量的内存,但是却不需要写到checkpoint里?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the saver checkpoint path and the memory accounting behind ps_memory_m and placement for the embedding-table model. Compare the memory reported for each server with the data actually serialized in its checkpoint; done means explaining the difference and identifying which memory is excluded from checkpoint output.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.