alipay / alipay/PainlessInferenceAcceleration
lookahead with do_sample=True does not take temperature, top_k, top_p
- Langage dominant
- Python
- Étoiles
- 370
- Forks
- 23
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
Here `lookahead_generation` doesn't take `logits_warper` as input:
https://github.com/alipay/PainlessInferenceAcceleration/blob/8015f12f7fe32acc102bb3eb51c4f8b3a420e79c/pia/lookahead/common/pretrained_model_batch.py#L426-L439
`logits_warper` is used in original `sample` to modify `next_tokens_scores`:
https://github.com/alipay/PainlessInferenceAcceleration/blob/8015f12f7fe32acc102bb3eb51c4f8b3a420e79c/pia/lookahead/common/pretrained_model_batch.py#L474-L486
and to modifies logits by temperature, top_k, top_p...
```python
if generation_config.temperature is not None and generation_config.temperature != 1.0:
warpers.append(TemperatureLogitsWarper(generation_config.temperature))
if generation_config.top_k is not None and generation_config.top_k != 0:
warpers.append(TopKLogitsWarper(top_k=generation_config.top_k, min_tokens_to_keep=min_tokens_to_keep))
if generation_config.top_p is not None and generation_config.top_p < 1.0:
warpers.append(TopPLogitsWarper(top_p=generation_config.top_p, min_tokens_to_keep=min_tokens_to_keep))
```
https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/generation/utils.py#L728-L733
This is not applied inside `lookahead_generation`. So with `do_sample=True` the temperature is always one
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.