huggingface / huggingface/lighteval
[BUG] Pipeline does not work with GenerationConfig
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
When setting `generation_config` in `transformers_model.TransformersModelConfig` it expects `generation_parameters` to be `None` (see [transformers_model.py#L188](https://github.com/huggingface/lighteval/blob/bfb1099ca82dc880baf1e90379378963cf7b2221/src/lighteval/models/transformers/transformers_model.py#L188)). However `Pipeline` expects `generation_parameters` always to be set (see [pipeline.py#L159](https://github.com/huggingface/lighteval/blob/bfb1099ca82dc880baf1e90379378963cf7b2221/src/lighteval/pipeline.py#L159)).
## To Reproduce
### Set `generation_config` only
```python
model_config = TransformersModelConfig(
pretrained=args.model,
dtype=args.dtype,
accelerator=accelerator,
max_gen_toks=args.max_new_tokens,
max_length=max_model_length,
use_chat_template=args.use_chat_template,
batch_size=args.batch_size,
generation_config=GenerationConfig(
max_new_tokens=args.max_new_tokens,
temperature=args.temperature,
top_p=args.top_p,
top_k=args.top_k,
repetition_penalty=args.repetition_penalty,
do_sample=True,
)
)
# Run
pipeline = Pipeline(
tasks=args.task,
pipeline_parameters=pipeline_params,
evaluation_tracker=evaluation_tracker,
model_config=model_config,
)
```
This will lead to following error:
```
Traceback (most recent call last):
File "test_script.py", line 170, in
main()
File "test_script.py", line 124, in main
pipeline = Pipeline(
^^^^^^^^^
File "test_script.py", line 159, in __init__
generation_parameters = asdict(model_config.generation_parameters) if model_config else {}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1273, in asdict
raise TypeError("asdict() should be called on dataclass instances")
TypeError: asdict() should be called on dataclass instances
```
### Set `generation_config` and `generation_parameters`
```python
model_config = TransformersModelConfig(
pretrained=args.model,
dtype=args.dtype,
accelerator=accelerator,
max_gen_toks=args.max_new_tokens,
max_length=max_model_length,
use_chat_template=args.use_chat_template,
batch_size=args.batch_size,
generation_parameters=GenerationParameters(max_new_tokens=args.max_new_tokens, seed=args.seed, temperature=args.temperature, top_p=args.top_p, top_k=args.top_k, repetition_penalty=args.repetition_penalty),
generation_config=GenerationConfig(
max_new_tokens=args.max_new_tokens,
temperature=args.temperature,
top_p=args.top_p,
top_k=args.top_k,
repetition_penalty=args.repetition_penalty,
do_sample=True if args.temperature > 0 else False,
)
)
# Run
pipeline = Pipeline(
tasks=args.task,
pipeline_parameters=pipeline_params,
evaluation_tracker=evaluation_tracker,
model_config=model_config,
)
```
This will lead to the error:
```
Can't use both generation_config and generation_parameters argument. Pass the generation parameters to your generation config object
```
## Expected behavior
The Pipeline class accepts a `model_config` with empty `generation_parameters`.
## Version info
0.8.1
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.