microsoft / microsoft/Graphormer

Binary pretrained model can't train a multi-class classifier?

Open
#172 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.5k
Forks
374
PR merge metrics
No merged PRs in 30d

Description

I use model_checkpoint = "clefourrier/graphormer-base-pcqm4mv2" # pre-trained model from which to fine-tune to be my pretrained model, and my dataset is here.
I follow the code in this website.
But change the

model = GraphormerForGraphClassification.from_pretrained(
    model_checkpoint, 

    # We have 167 attack patterns and 1 benign
    num_classes=168, 

    # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
    ignore_mismatched_sizes = True, 

Since I have 168 class to be classified.
Then when I run the python script, I encountered:

Some weights of GraphormerForGraphClassification were not initialized from the model checkpoint at clefourrier/graphormer-base-pcqm4mv2 and are newly initialized because the shapes did not match:
- classifier.classifier.weight: found shape torch.Size([1, 768]) in the checkpoint and torch.Size([168, 768]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
  0%|                                                                                                                                                                                                                                          | 0/23120 [00:00<?, ?it/s]

For about 20 min, the log didn't move at all.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the GraphormerForGraphClassification.from_pretrained call and the graphml-classification.md example, then reproduce the Python script using the clefourrier/graphormer-base-pcqm4mv2 checkpoint and 168 classes. Check whether the classifier-size warning or the stalled training progress is the issue to address; done means the multi-class training run proceeds or the failure is clearly explained.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.