AI-Hypercomputer / AI-Hypercomputer/maxtext

Support beam search

Abierto
#594 0 comentarios 0 reacciones 1 asignado Reclamado por @vipannalla Ver en GitHub
feature request inference
Lenguaje dominante
Python
Estrellas
2.4k
Forks
607
Merge medio
2 d 19 h
PR fusionados (30 d)
158

Descripción

Hi,

It would be nice to support beam search.

There is [the reference flax implementation in wmt example](https://github.com/google/flax/blob/main/examples/wmt/decode.py) and [the equivalent one from `transformers`](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/flax_utils.py).

I am guessing that we could:
* duplicate inputs per num_beams initially
* at each step we do:
* decode_step
* select top beams
* overwrite entire past cache per selected beams
* update cache with new selected tokens

So maybe the extra step here is to add the "overwrite entire past cache per selected beams"?
Curious if you have suggestions for implementation

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.