AI-Hypercomputer / AI-Hypercomputer/maxtext

Support beam search

未关闭
#594 0 条评论 0 个 reaction 已指派 1 人 已被 @vipannalla 认领 在 GitHub 查看
feature request inference
主要语言
Python
星标
2.4k
派生
607
平均合并
2 天 19 小时
30 天内合并 PR
158

描述

Hi,

It would be nice to support beam search.

There is [the reference flax implementation in wmt example](https://github.com/google/flax/blob/main/examples/wmt/decode.py) and [the equivalent one from `transformers`](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/flax_utils.py).

I am guessing that we could:
* duplicate inputs per num_beams initially
* at each step we do:
* decode_step
* select top beams
* overwrite entire past cache per selected beams
* update cache with new selected tokens

So maybe the extra step here is to add the "overwrite entire past cache per selected beams"?
Curious if you have suggestions for implementation

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。