awslabs / awslabs/graphstorm

Save BERT model separately

Open
#564 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
452
Forks
76
PR merge metrics
No merged PRs in 30d

Description

Currently, we save all model parameters together. This leads to a problem that if the code is changed, we will not be able to load the model. What's even worse is that the failure happens silently.

This problem becomes more serious when we need to fine-tune the BERT model. It is common that we need to fine-tune the BERT model and load the fine-tuned BERT model to train GNN model. It is expensive to fine-tune the BERT model. Once it is tuned, we prefer to keep using it even if the GraphStorm code is changed.

To solve this problem, we should save the BERT model separately from the remaining model parameters.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named in the issue. Start by locating the model serialization and BERT fine-tuning load paths, then trace how the remaining parameters are saved. Done means BERT is stored separately, can be reused after GraphStorm code changes, and loading failures are no longer silent.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.