[Feature] Multi-variable prompt templates
- Dominant language
- Python
- Stars
- 291
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
I see from [this comment](https://github.com/aws/fmeval/issues/205#issuecomment-1987712759) this feature may be coming already, but the attached issue was closed with the interim workaround.
For our use-case we have a multi-field dataset for example:
| source_doc | question | ref_answers
|:---|:---|:---|
| (full text) | What date was the agreement signed, in YYYY-MM-DD format? | 2024-04-09 |
| ... | ... | ... |
...Where the final LLM prompt would combine both the source document and the question, in a (constant) template. I could easily see more general use-cases with other fields too.
Today, we're hacking around fmeval by doing prompt fulfilment as a separate step before the library. It would be much better if fmeval is able to directly process raw multi-field datasets like this, by taking a prompt template that can reference arbitrary fields from the source record.
My reason for revisiting this was Claude v3's messages API, which means we're going to have to do more sophisticated fulfilment on our side to achieve the same effect.
Contributor guide
Research direction
Start with the linked comment and issue #205 to understand the interim workaround and any prior design discussion. Then trace how fmeval currently performs prompt fulfilment and consumes source records; done should allow a constant template to reference arbitrary fields in raw multi-field datasets, including the described source_doc and question example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100