huggingface / huggingface/lighteval

[FT] Add System Prompt field in LightevalTaskConfig that can be used by model clients

Open
#410 14 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered
My models currently don't follow the template I give. I want to give a system prompt that nudges the models to provide output the way I want it.

## Solution/Feature
We could add a new field in LightevalTaskConfig that can be consumed by model clients like LiteLLMClient to send to the API.

## Possible alternatives
Alternatively, we could use the generation_grammar, but I think this may be a bit overkill and may be more difficult to implement for different API providers.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.