EleutherAI / EleutherAI/steering-llama3

Wrong steering behavior in comparison with equation in paper

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
30
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Hi authors,

I recently came across your paper "Refusal in LLMs is an Affine Function" and am very interested in your work. I decided to reproduce the results. However, while debugging your code, I noticed that the steering function in your `steer.py` file doesn't seem to match your described method.

```Python
def hook(model, input, output):
# with record_function("steer_hook"):
# with record_function("steer"):
# print(type(model.mlp), type(model.self_attn))
# print("output", output[0].device, output[0].dtype, output[0].shape)
output[0][..., self.start:self.end, :] += u
return output
```

This hook function appears to be implementing Contrastive Activation Addition (CAA) rather than your proposed Affine Concept Editing (ACE) method.

Would it be possible to get the correct implementation? Thank you very much!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in steer.py at the hook function shown in the issue, then compare its activation update with the Affine Concept Editing equation in the cited paper. Run the existing reproduction workflow to inspect the current steering behavior and determine whether the implementation matches the paper's method. Done means the code uses the described ACE behavior rather than the currently observed CAA-style update.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.