GPT-neo generate is ignoring passed position ids

System Info

python version: 3.9
transformers version: 4.18

Who can help?

@patil-suraj @patrickvonplaten

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch 

text = "hi there: "
modelname = "EleutherAI/gpt-neo-125M"
model = AutoModelForCausalLM.from_pretrained(modelname)
tokenizer = AutoTokenizer.from_pretrained(modelname)

inputs = tokenizer(text, return_tensors ="pt")
inputs["position_ids"] = inputs["attention_mask"].cumsum(-1)
print(inputs)
a = model.generate(**inputs)


inputs["position_ids"] = inputs["position_ids"] + 10
print(inputs)
b = model.generate(**inputs)

# a and b should be different because the position ids are different
print(a)
print(b)

the result

a = tensor([[5303,  612,   25,  220,  220,  220,  220,  220,  220,  220,  220,  220,
          220,  220,  220,  220,  220,  220,  220,  220]])
b = tensor([[5303,  612,   25,  220,  220,  220,  220,  220,  220,  220,  220,  220,
          220,  220,  220,  220,  220,  220,  220,  220]])

Expected behavior

The outputs a and b should (almost always) be different because different position ids should be passed to the model's forward function, resulting in different activations. The issue seems to be here: https://github.com/huggingface/transformers/blob/ee393c009a243bbb86fa11d2efa771a1704d85cf/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L699

Specifically, the else statement is such that if both an attention mask and position ids are passed, the position ids are erased. In such a scenario, the default position ids in the model's forward function (https://github.com/huggingface/transformers/blob/ee393c009a243bbb86fa11d2efa771a1704d85cf/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L537) are used, rather than the passed-in position ids. 

Proposed fix: remove the `else` block on line 699 and unindent the `if past` block.

Start with src/transformers/models/gpt_neo/modeling_gpt_neo.py at the position-id handling around lines 537 and 699, then run the reproduction script from the issue with GPT-Neo 125M. Trace how attention_mask, position_ids, and past are passed through generate and forward; done means differing position IDs affect the generated outputs as expected.