EleutherAI / EleutherAI/steering-llama3
Wrong steering behavior in comparison with equation in paper
- Dominant language
- Python
- Stars
- 30
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
Hi authors,
I recently came across your paper "Refusal in LLMs is an Affine Function" and am very interested in your work. I decided to reproduce the results. However, while debugging your code, I noticed that the steering function in your `steer.py` file doesn't seem to match your described method.
```Python
def hook(model, input, output):
# with record_function("steer_hook"):
# with record_function("steer"):
# print(type(model.mlp), type(model.self_attn))
# print("output", output[0].device, output[0].dtype, output[0].shape)
output[0][..., self.start:self.end, :] += u
return output
```
This hook function appears to be implementing Contrastive Activation Addition (CAA) rather than your proposed Affine Concept Editing (ACE) method.
Would it be possible to get the correct implementation? Thank you very much!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in steer.py at the hook function shown in the issue, then compare its activation update with the Affine Concept Editing equation in the cited paper. Run the existing reproduction workflow to inspect the current steering behavior and determine whether the implementation matches the paper's method. Done means the code uses the described ACE behavior rather than the currently observed CAA-style update.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100