awslabs / awslabs/graphstorm

Learnable embeddings are not cleaned up when using top-k model parameter (related to SageMaker HPO)

Open
#1,252 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
452
Forks
76
PR merge metrics
No merged PRs in 30d

Description

Currently, when a user sets a specific number of top-k models to be retained, the model files are removed but the learnable embeddings of every epoch are maintained.

The expected behavior is that only the embeddings for the topk best models should be retained.

Here's some example output from a SageMaker training job:

```
model_checkpoint $ tree
.
├── epoch-0
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
├── epoch-1
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
├── epoch-29
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
├── epoch-3
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
├── epoch-30
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
├── epoch-36
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
├── epoch-4
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
├── epoch-43
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
├── epoch-45
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
├── epoch-5
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
├── epoch-51
│ ├── author
│ │ ├── sparse_emb_00000.pt
│ │ └── sparse_emb_00001.pt
│ ├── model.bin
│ ├── optimizers.bin
│ ├── paper
│ │ ├── sparse_emb_00000.pt
│ │ └── sparse_emb_00001.pt
│ └── subject
│ ├── sparse_emb_00000.pt
│ └── sparse_emb_00001.pt
├── epoch-6
│ ├── author
│ │ └── sparse_emb_00001.pt
│ ├── paper
│ │ └── sparse_emb_00001.pt
│ └── subject
│ └── sparse_emb_00001.pt
```

Contributor guide

Open the contributing guide

Research direction

The issue names no source files or tests. Start by reproducing the SageMaker training-job checkpoint layout with a top-k setting, then trace the retention path for model files and learnable embeddings. Done means only embeddings belonging to the retained top-k models remain; add a focused regression test if the repository provides one.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.