EleutherAI / EleutherAI/knowledge-neurons
grad error using KnowledgeNeurons with model_name = "gpt2"
- Dominant language
- Python
- Stars
- 159
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
When I initialize KnowledgeNeurons with model_name = 'gpt2', I get an RuntimeError when trying to run kn.get_refined_neurons()
The following snippet would be able to reproduce the error on a colab notebook:
```
!pip install knowledge-neurons
!nvidia-smi
from knowledge_neurons import (
KnowledgeNeurons,
initialize_model_and_tokenizer,
model_type,
)
import random
import torch
import torch.nn.functional as F
# setup model, tokenizer + kn class
MODEL_NAME = "gpt2"
model, tokenizer = initialize_model_and_tokenizer(MODEL_NAME)
kn = KnowledgeNeurons(model, tokenizer,model_type(MODEL_NAME))
TEXT = "Sarah was visiting [MASK], the capital of france"
GROUND_TRUTH = "paris"
BATCH_SIZE = 10
STEPS = 20
ENG_TEXTS = [
"Sarah was visiting [MASK], the capital of france",
"The capital of france is [MASK]",
"[MASK] is the capital of france",
"France's capital [MASK] is a hotspot for romantic vacations",
"The eiffel tower is situated in [MASK]",
"[MASK] is the most populous city in france",
"[MASK], france's capital, is one of the most popular tourist destinations in the world",
]
FRENCH_TEXTS = [
"Sarah visitait [MASK], la capitale de la france",
"La capitale de la france est [MASK]",
"[MASK] est la capitale de la france",
"La capitale de la France [MASK] est un haut lieu des vacances romantiques",
"La tour eiffel est située à [MASK]",
"[MASK] est la ville la plus peuplée de france",
"[MASK], la capitale de la france, est l'une des destinations touristiques les plus prisées au monde",
]
TEXTS = ENG_TEXTS + FRENCH_TEXTS
refined_neurons_eng = kn.get_refined_neurons(
ENG_TEXTS,
GROUND_TRUTH,
p=P,
batch_size=BATCH_SIZE,
steps=STEPS,
)
refined_neurons_fr = kn.get_refined_neurons(
FRENCH_TEXTS,
GROUND_TRUTH,
p=P,
batch_size=BATCH_SIZE,
steps=STEPS,
)
refined_neurons = kn.get_refined_neurons(
TEXTS,
GROUND_TRUTH,
p=P,
batch_size=BATCH_SIZE,
steps=STEPS,
)
```
Given below is the full traceback:
```Getting coarse neurons for each prompt...: 0%| | 0/7 [00:00 in ()
45 p=P,
46 batch_size=BATCH_SIZE,
---> 47 steps=STEPS,
48 )
49 refined_neurons_fr = kn.get_refined_neurons(
5 frames
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in _make_grads(outputs, grads)
49 if out.requires_grad:
50 if out.numel() != 1:
---> 51 raise RuntimeError("grad can be implicitly created only for scalar outputs")
52 new_grads.append(torch.ones_like(out, memory_format=torch.preserve_format))
53 else:
RuntimeError: grad can be implicitly created only for scalar outputs
```
Grad Error when initializing for the "gpt2" model
@StellaAthena @sdtblck also what should be the input and the target for "gpt2" model given it's autoregressive mechanism??
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.