gemma3 with deepspeed zero 3

Issue

[rank0]: self.initialize_weights()
[rank0]: File "/data/scy/anaconda3/envs/pytorch_2.7.1/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/scy/anaconda3/envs/pytorch_2.7.1/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2984, in initialize_weights
[rank0]: self.smart_apply(self._initialize_weights)
[rank0]: File "/data/scy/anaconda3/envs/pytorch_2.7.1/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2975, in smart_apply
[rank0]: module.smart_apply(module._initialize_weights)
[rank0]: File "/data/scy/anaconda3/envs/pytorch_2.7.1/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2975, in smart_apply
[rank0]: module.smart_apply(module._initialize_weights)
[rank0]: File "/data/scy/anaconda3/envs/pytorch_2.7.1/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2977, in smart_apply
[rank0]: module.smart_apply(fn)
[rank0]: File "/data/scy/anaconda3/envs/pytorch_2.7.1/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2978, in smart_apply
[rank0]: fn(self)
[rank0]: File "/data/scy/anaconda3/envs/pytorch_2.7.1/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2952, in _initialize_weights
[rank0]: self._init_weights(module)
[rank0]: File "/data/scy/anaconda3/envs/pytorch_2.7.1/lib/python3.12/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 434, in _init_weights
[rank0]: super()._init_weights(module)
[rank0]: File "/data/scy/anaconda3/envs/pytorch_2.7.1/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2929, in init_weights
[rank0]: module.weight.data[module.padding_idx].zero()
[rank0]: ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
[rank0]: IndexError: index 0 is out of bounds for dimension 0 with size 0

Environment
absl-py 2.3.1
accelerate 1.11.0
addict 2.4.0
aiofiles 24.1.0
aiohappyeyeballs 2.6.1
aiohttp 3.13.2
aiosignal 1.4.0
aliyun-python-sdk-core 2.16.0
aliyun-python-sdk-kms 2.16.5
annotated-doc 0.0.3
annotated-types 0.7.0
antlr4-python3-runtime 4.13.2
anyio 4.11.0
astor 0.8.1
asttokens 3.0.0
attrdict 2.0.1
attrs 25.4.0
av 16.0.1
binpacking 1.5.2
black 24.10.0
blake3 1.0.8
boto3 1.40.61
botocore 1.40.61
brotli 1.1.0
cachetools 6.2.1
cbor2 5.7.1
certifi 2025.10.5
cffi 2.0.0
charset-normalizer 3.4.4
click 8.2.1
cloudpickle 3.1.1
colorama 0.4.6
colorlog 6.10.1
compressed-tensors 0.11.0
contourpy 1.3.3
cpm-kernels 1.0.11
crcmod 1.7
cryptography 46.0.3
cupy-cuda12x 13.6.0
cycler 0.12.1
dacite 1.9.2
datasets 3.6.0
decorator 5.2.1
decord 0.6.0
deepspeed 0.18.1
depyf 0.19.0
dill 0.3.8
diskcache 5.6.3
distro 1.9.0
dnspython 2.8.0
docstring-parser 0.17.0
dotenv 0.9.9
einops 0.8.1
email-validator 2.3.0
et-xmlfile 2.0.0
evalscope 1.1.1
evaluate 0.4.6
executing 2.2.1
fastapi 0.120.1
fastapi-cli 0.0.14
fastapi-cloud-cli 0.3.1
fastrlock 0.8.3
ffmpy 0.6.4
filelock 3.20.0
fire 0.7.1
flake8 7.1.1
flash-attn 2.8.3
fonttools 4.60.1
frozendict 2.4.6
frozenlist 1.8.0
fsspec 2025.3.0
func-timeout 4.3.5
future 1.0.0
fuzzywuzzy 0.18.0
gguf 0.17.1
gitdb 4.0.12
gitpython 3.1.45
google-auth 2.42.0
google-genai 1.46.0
gradio 5.49.1
gradio-client 1.13.3
groovy 0.1.2
grpcio 1.76.0
h11 0.16.0
h5py 3.15.1
hf-xet 1.2.0
hjson 3.1.0
httpcore 1.0.9
httptools 0.7.1
httpx 0.28.1
huggingface-hub 0.36.0
human-eval 1.0.3
idna 3.11
imageio 2.37.0
immutabledict 4.2.2
importlib-metadata 8.7.0
iniconfig 2.3.0
interegular 0.3.3
ipdb 0.13.13
ipython 9.6.0
ipython-pygments-lexers 1.1.1
jedi 0.19.2
jieba 0.42.1
jinja2 3.1.6
jiter 0.11.1
jmespath 0.10.0
joblib 1.5.2
json-repair 0.52.3
json5 0.12.1
jsonlines 4.0.0
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
kiwisolver 1.4.9
langdetect 1.0.9
lark 1.2.2
latex2sympy2-extended 1.0.6
levenshtein 0.27.1
liger-kernel 0.6.3
llguidance 0.7.30
llvmlite 0.44.0
lm-format-enforcer 0.11.3
lxml 6.0.2
markdown 3.9
markdown-it-py 4.0.0
markupsafe 3.0.3
math-verify 0.5.2
matplotlib 3.10.7
matplotlib-inline 0.2.1
mccabe 0.7.0
mdurl 0.1.2
mistral-common 1.8.5
mmengine-lite 0.10.7
modelscope 1.31.0
mpmath 1.3.0
ms-opencompass 0.1.6
ms-swift 3.10.0.dev1 /work/HHRI-AI/POC/public/Ms-swiff-Training/trivita-llm/3rdparty/ms-swift
ms-vlmeval 0.0.18
msgpack 1.1.2
msgspec 0.19.0
multidict 6.7.0
multiprocess 0.70.16
mypy-extensions 1.1.0
networkx 3.5
ninja 1.13.0
nltk 3.9.2
numba 0.61.2
numpy 1.26.4
nvidia-cublas-cu12 12.8.4.1
nvidia-cuda-cupti-cu12 12.8.90
nvidia-cuda-nvrtc-cu12 12.8.93
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cudnn-cu12 9.10.2.21
nvidia-cufft-cu12 11.3.3.83
nvidia-cufile-cu12 1.13.1.3
nvidia-curand-cu12 10.3.9.90
nvidia-cusolver-cu12 11.7.3.90
nvidia-cusparse-cu12 12.5.8.93
nvidia-cusparselt-cu12 0.7.1
nvidia-ml-py 13.580.82
nvidia-nccl-cu12 2.27.3
nvidia-nvjitlink-cu12 12.8.93
nvidia-nvtx-cu12 12.8.90
openai 2.6.1
openai-harmony 0.0.4
opencc 1.1.9
opencv-python 4.11.0.86
opencv-python-headless 4.11.0.86
openpyxl 3.1.5
orjson 3.11.4
oss2 2.19.1
outlines-core 0.2.11
overrides 7.7.0
packaging 25.0
pandas 2.3.3
parso 0.8.5
partial-json-parser 0.2.1.1.post6
pathspec 0.12.1
peft 0.17.1
pexpect 4.9.0
pillow 11.3.0
platformdirs 4.5.0
pluggy 1.6.0
portalocker 3.2.0
prettytable 3.16.0
prometheus-client 0.23.1
prometheus-fastapi-instrumentator 7.1.0
prompt-toolkit 3.0.52
propcache 0.4.1
protobuf 6.33.0
psutil 7.1.2
ptyprocess 0.7.0
pure-eval 0.2.3
py-cpuinfo 9.0.0
pyarrow 22.0.0
pyasn1 0.6.1
pyasn1-modules 0.4.2
pybase64 1.4.2
pycodestyle 2.12.1
pycountry 24.6.1
pycparser 2.23
pycryptodome 3.23.0
pydantic 2.11.10
pydantic-core 2.33.2
pydantic-extra-types 2.10.6
pydub 0.25.1
pyecharts 2.0.9
pyflakes 3.2.0
pygments 2.19.2
pyparsing 3.2.5
pypinyin 0.55.0
pytest 8.4.2
python-dateutil 2.9.0.post0
python-dotenv 1.2.1
python-json-logger 4.0.0
python-levenshtein 0.27.1
python-multipart 0.0.20
pytz 2025.2
pyyaml 6.0.3
pyzmq 27.1.0
qwen-vl-utils 0.0.14
rank-bm25 0.2.2
rapidfuzz 3.14.1
ray 2.51.0
referencing 0.37.0
regex 2025.10.23
requests 2.32.5
rich 13.9.4
rich-toolkit 0.15.1
rignore 0.7.1
rouge 1.0.1
rouge-chinese 1.0.3
rouge-score 0.1.2
rpds-py 0.28.0
rsa 4.9.1
ruff 0.14.2
s3transfer 0.14.0
sacrebleu 2.5.1
safehttpx 0.1.7
safetensors 0.6.2
scikit-learn 1.7.2
scipy 1.16.3
seaborn 0.13.2
semantic-version 2.10.0
sentence-transformers 5.1.2
sentencepiece 0.2.1
sentry-sdk 2.42.1
setproctitle 1.3.7
setuptools 79.0.1
shellingham 1.5.4
simplejson 3.20.2
six 1.17.0
smmap 5.0.2
sniffio 1.3.1
sortedcontainers 2.4.0
soundfile 0.13.1
soxr 1.0.0
stack-data 0.6.3
starlette 0.49.1
sty 1.0.6
swanlab 0.6.13
sympy 1.14.0
tabulate 0.9.0
tenacity 9.1.2
tensorboard 2.20.0
tensorboard-data-server 0.7.2
termcolor 3.2.0
threadpoolctl 3.6.0
tiktoken 0.12.0
timeout-decorator 0.5.0
timm 1.0.21
tokenizers 0.22.1
tomlkit 0.13.3
torch 2.8.0
torchaudio 2.8.0
torchvision 0.23.0
tqdm 4.67.1
traitlets 5.14.3
transformers 4.57.1
transformers-stream-generator 0.0.5
triton 3.4.0
trivita-llm 0.0.1 /work/HHRI-AI/POC/public/Ms-swiff-Training/trivita-llm
trl 0.23.1
typer 0.20.0
typing-extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.2
urllib3 2.5.0
uvicorn 0.38.0
uvloop 0.22.1
validators 0.35.0
vllm 0.11.0
wandb 0.22.3
watchfiles 1.1.1
wcwidth 0.2.14
websockets 15.0.1
werkzeug 3.1.3
word2number 1.1
wrapt 2.0.0
xformers 0.0.32.post1
xgrammar 0.1.25
xlsxwriter 3.2.9
xxhash 3.6.0
yapf 0.43.0
yarl 1.22.0
zipp 3.23.0
zstandard 0.25.0

Start by reproducing Gemma3 initialization with DeepSpeed ZeRO-3 using the traceback entry point in transformers/models/gemma3/modeling_gemma3.py. Trace the call into transformers/modeling_utils.py, especially initialize_weights and init_weights, to identify why the indexed weight has size zero. Done means Gemma3 initializes under ZeRO-3 without the reported IndexError.