huggingface / huggingface/lighteval
[FT] Assistant Response Prefilling
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Solution/Feature
Context:
[docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response)
[anthropic.com/news/claude-2-1-prompting](https://www.anthropic.com/news/claude-2-1-prompting)
I would love to be able to manually inject assistant responses during eval to perform some ablations. Ideally, I would be able to modify the messages like such:
```
messages = [
{
"role": "user",
"content": prompt,
},
"role": "assistant",
"content": 'According to ',
}
]
```
And it should use this to generate up through assistant's response but not ending it, so the model continues.
## Possible alternatives
Adopting a custom chat template could work, but it would be nice to see this feature integrated as I anticipate a lot of researchers will have need for easy access to this.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.