inference model can not see anything useful

my env:
cuda 11.8
torch 2.5.1
torchvision 0.19.0

inference code:
```
with torch.no_grad():
# run image encoder to get the image embeddings
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# run the model to get the response
outputs = vl_gpt.generate(
input_ids=prepare_inputs["input_ids"],
inputs_embeds=inputs_embeds,
attention_mask=prepare_inputs.attention_mask,
pad_token_id=tokenizer.eos_token_id,
bos_token_id=tokenizer.bos_token_id,
eos_token_id=tokenizer.eos_token_id,
max_new_tokens=1024,

do_sample=False,
# repetition_penalty=1.1,

# do_sample=True,
# temperature=1.0,
# top_p=0.9,
# repetition_penalty=1.1,

use_cache=True,
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=False)
print(f"{prepare_inputs['sft_format'][0]}", answer)

vg_image = parse_ref_bbox(answer, image=pil_images[0])
if vg_image is not None:
vg_image.save("./vg.jpg", format="JPEG", quality=85)
```

output:
```

vg_image: None

<|User|>:
<|ref|>The giraffe at the back.<|/ref|>.

<|Assistant|>: <｜begin▁of▁sentence｜><|User|>:
<|ref|>The giraffe at the back.<|/ref|>.

<|Assistant|>:<｜end▁of▁sentence｜>

```