AI-Hypercomputer / AI-Hypercomputer/maxtext

Gemma3 to hf mapping error

Ouverte
#2,867 5 commentaires 0 réactions 1 personne assignée Réclamée par @hengtaoguo Voir sur GitHub
bug
Langage dominant
Python
Étoiles
2.4k
Forks
607
Merge moyen
2 j 19 h
PR mergées (30 j)
158

Description

### Bug report

After finetuning the model, I used the following command to convert my MaxText Gemma3 Checkpoint to HuggingFace format:

```bash
python3 -m MaxText.utils.ckpt_conversion.to_huggingface src/MaxText/configs/base.yml \ model_name='gemma3-4b' \
hf_access_token={hf token} \
load_parameters_path={my checkpoint path} \
base_output_directory=/tmp/gemma3-4B-cpt-hf \
use_multimodal=false # tried both false & true \
scan_layers=false # cannot create scanned checkpoint (another bug)
```

For testing purposes, I set up a vllm server on the TPU

```bash
pip3.11 install vllm-tpu

vllm serve /tmp/gemma3-4B-cpt-hf \
--disable-log-requests \
--tensor_parallel_size=4 \
--api-key {my api key}
```

Error message from vllm
```
ValueError: Gemma3 uses `gelu_pytorch_tanh` as the hidden activation function. Please set `hidden_act` and `hidden_activation` to `gelu_pytorch_tanh`.
```

[MaxText Conversion config](https://github.com/AI-Hypercomputer/maxtext/blob/c2574abeaad6e481d1ee62f961080f86b662a2dc/src/MaxText/utils/ckpt_conversion/utils/hf_model_configs.py#L38C31-L38C35) uses `"hidden_activation": "gelu"` for text generation

[huggingface config for text generation](https://huggingface.co/docs/transformers/v4.55.4/model_doc/gemma3#transformers.Gemma3TextConfig.hidden_activation) uses `gelu_pytorch_tanh`

[vllm](https://github.com/vllm-project/vllm/blob/c02a2705f9ceeb00b5d32453621f997b2ceafbea/vllm/model_executor/models/gemma3.py#L90) explicitly check for `gelu_pytorch_tanh` which resulted in the error above

### Logs/Output

### Environment Information

TPU creation command
```bash
export TPU_COUNT=4
gcloud alpha compute tpus tpu-vm create $TPU_ID \
--zone="${ZONE}" \
--accelerator-type="v6e-${TPU_COUNT}" \
--version=v2-alpha-tpuv6e \
--spot \
--service-account=${service_account}
```

Tried installation from source and uv as suggested in the documentation of MaxText.

### Additional Context

_No response_

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.