FlanT5 training and zero tensors

Hi, I'm training a FlanT5 network. The training completes successfully, but when I try to run a simple inference, I have a tensor of zeros, so the prediction is null.

Example:

tokenizer = AutoTokenizer.from_pretrained(path, use_fast=False)
model = T5ForConditionalGeneration.from_pretrained(path, low_cpu_mem_usage=True, torch_dtype=torch.float16).cuda()

tokenized_text = tokenizer(query, return_tensors="pt")

source_ids = tokenized_text["input_ids"].to(device, dtype=torch.long)

generated_ids = model.generate(input_ids=source_ids)

Output:

tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
       device='cuda:0')

I tried to run several trainings, both on flanT5-xl and flanT5-large, both on my personal dataset and a dummy.json dataset.

That's my training configuration:

!python3 -m torch.distributed.run --nproc_per_node=6 fastchat/train/train_flant5.py \
    --model_name_or_path google/flan-t5-xl \
    --data_path playground/data/dummy.json \
    --fp16 True \
    --output_dir ./output \
    --num_train_epochs 5 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps 2 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 99999 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp_transformer_layer_cls_to_wrap T5Block \
    --tf32 False \
    --fsdp "full_shard auto_wrap" \
    --model_max_length 256 \
    --gradient_checkpointing True \
    --preprocessed_path ./preprocessed_data/processed.json 

Any idea what's going on? Thank you.