Compatibility Issue with Prefix-Tuned Models using PEFT in FastChat

Nobody has claimed this yet.

I am encountering a compatibility issue when trying to use a prefix-tuned model with FastChat. The model was fine-tuned using the PEFT library, which utilizes `past_key_values` for injecting prefixes. This conflicts with FastChat's streaming functionality in `fastchat.serve.inference.generate_stream`, which also relies on `past_key_values`.

To illustrate the problem, I have created a fork with a GPT2 model integrated with a random prefix adapter, which can be found here:
https://github.com/fzalkow/FastChat/blob/test_prefix/fastchat/model/model_adapter.py#L2271-L2298

I launched the FastChat server and API with these commands:

```
python3 -m fastchat.serve.controller --host 127.0.0.1
python3 -m fastchat.serve.model_worker --host 127.0.0.1 --controller-address http://127.0.0.1:21001 --model-path openai-community/gpt2
python3 -m fastchat.serve.openai_api_server --host 127.0.0.1 --controller-address http://127.0.0.1:21001 --port 8000
```

When I make an API call:

```
curl http://127.0.0.1:8000/v1/completions -H "Content-Type: application/json" -d '{"model": "gpt2", "prompt": "Does fastchat work with prefix-tuned models?", "max_tokens": 1014, "temperature": 0.0}'
```

I receive an `Internal Server Error` due to the `got multiple values for keyword argument 'past_key_values'` error.

The prefix-tuned model's forward call from the PEFT library uses the `past_key_values` keyword, which is here:
https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py#L1126-L1127

And FastChat's use of the same keyword in its streaming functionality is here:
https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/inference.py#L160-L167

Does anyone know how to resolve this conflict? I currently use a dirty workaround to get it working, where I reimplemented `generate_stream` to use `model.generate` in a single step, but this bypasses the entire streaming functionality, which is not ideal. Any suggestions or guidance would be greatly appreciated.

Reproduce the failure with the provided FastChat controller, model worker, API server, and curl commands. Read fastchat/serve/inference.py around lines 160-167 and the linked model_adapter.py and PEFT forward-call references to trace the duplicate past_key_values arguments. Done means prefix-tuned models work through the normal streaming API without the duplicate-keyword error while preserving streaming behavior.

Written by the indexing model from the issue text.