alipay / alipay/PainlessInferenceAcceleration

lookahead with do_sample=True does not take temperature, top_k, top_p

未关闭
#26 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
370
派生
23
PR 合并指标
30 天内没有已合并 PR

描述

Here `lookahead_generation` doesn't take `logits_warper` as input:

https://github.com/alipay/PainlessInferenceAcceleration/blob/8015f12f7fe32acc102bb3eb51c4f8b3a420e79c/pia/lookahead/common/pretrained_model_batch.py#L426-L439

`logits_warper` is used in original `sample` to modify `next_tokens_scores`:

https://github.com/alipay/PainlessInferenceAcceleration/blob/8015f12f7fe32acc102bb3eb51c4f8b3a420e79c/pia/lookahead/common/pretrained_model_batch.py#L474-L486

and to modifies logits by temperature, top_k, top_p...

```python
if generation_config.temperature is not None and generation_config.temperature != 1.0:
warpers.append(TemperatureLogitsWarper(generation_config.temperature))
if generation_config.top_k is not None and generation_config.top_k != 0:
warpers.append(TopKLogitsWarper(top_k=generation_config.top_k, min_tokens_to_keep=min_tokens_to_keep))
if generation_config.top_p is not None and generation_config.top_p < 1.0:
warpers.append(TopPLogitsWarper(top_p=generation_config.top_p, min_tokens_to_keep=min_tokens_to_keep))
```

https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/generation/utils.py#L728-L733

This is not applied inside `lookahead_generation`. So with `do_sample=True` the temperature is always one

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。