alipay / alipay/PainlessInferenceAcceleration
lookahead with do_sample=True does not take temperature, top_k, top_p
- Ngôn ngữ chính
- Python
- Star
- 370
- Fork
- 23
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
Here `lookahead_generation` doesn't take `logits_warper` as input:
https://github.com/alipay/PainlessInferenceAcceleration/blob/8015f12f7fe32acc102bb3eb51c4f8b3a420e79c/pia/lookahead/common/pretrained_model_batch.py#L426-L439
`logits_warper` is used in original `sample` to modify `next_tokens_scores`:
https://github.com/alipay/PainlessInferenceAcceleration/blob/8015f12f7fe32acc102bb3eb51c4f8b3a420e79c/pia/lookahead/common/pretrained_model_batch.py#L474-L486
and to modifies logits by temperature, top_k, top_p...
```python
if generation_config.temperature is not None and generation_config.temperature != 1.0:
warpers.append(TemperatureLogitsWarper(generation_config.temperature))
if generation_config.top_k is not None and generation_config.top_k != 0:
warpers.append(TopKLogitsWarper(top_k=generation_config.top_k, min_tokens_to_keep=min_tokens_to_keep))
if generation_config.top_p is not None and generation_config.top_p < 1.0:
warpers.append(TopPLogitsWarper(top_p=generation_config.top_p, min_tokens_to_keep=min_tokens_to_keep))
```
https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/generation/utils.py#L728-L733
This is not applied inside `lookahead_generation`. So with `do_sample=True` the temperature is always one
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.