microsoft / microsoft/Graphormer
Binary pretrained model can't train a multi-class classifier?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 374
- PR merge metrics
- No merged PRs in 30d
Description
I use model_checkpoint = "clefourrier/graphormer-base-pcqm4mv2" # pre-trained model from which to fine-tune
to be my pretrained model, and my dataset is here.
I follow the code in this website.
But change the
model = GraphormerForGraphClassification.from_pretrained(
model_checkpoint,
# We have 167 attack patterns and 1 benign
num_classes=168,
# provide this in case you're planning to fine-tune an already fine-tuned checkpoint
ignore_mismatched_sizes = True,
Since I have 168 class to be classified.
Then when I run the python script, I encountered:
Some weights of GraphormerForGraphClassification were not initialized from the model checkpoint at clefourrier/graphormer-base-pcqm4mv2 and are newly initialized because the shapes did not match:
- classifier.classifier.weight: found shape torch.Size([1, 768]) in the checkpoint and torch.Size([168, 768]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
0%| | 0/23120 [00:00<?, ?it/s]
For about 20 min, the log didn't move at all.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the GraphormerForGraphClassification.from_pretrained call and the graphml-classification.md example, then reproduce the Python script using the clefourrier/graphormer-base-pcqm4mv2 checkpoint and 168 classes. Check whether the classifier-size warning or the stalled training progress is the issue to address; done means the multi-class training run proceeds or the failure is clearly explained.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100