RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

Hi
I met 2 issues when running the demo script:
The first issue occured when importing "videollama2.constants" is "Failed to import transformers.models.llama.modeling_llama", and I fixed it by "pip install flash-attn --no-build-isolation --no-cache-dir".

The second is what mention in title "RuntimeError: probability tensor contains either `inf`, `nan` or element < 0" is during running the last step that:
- model: VideoLLaMA2-7B
- video: both sora.mp4 and cat_and_chicken.mp4
- enviroment: ubuntu 22.04, python 3.11, cuda 11.8, GPU RTX4070 12G

I had tried to:
1. switch model.dtype to bfloat16, but still met the error
2. set do_sample=False, it worked but the output was null (and it took 30 minutes to process a 20 second video, I guessed this might due to my gpu device...)

Do you have any ideas to solve ? much thanks

------------------------------------------------------------------------------------------
{
"name": "RuntimeError",
"message": "probability tensor contains either `inf`, `nan` or element < 0",
"stack": "---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[51], line 2
1 with torch.inference_mode():
----> 2 output_ids = model.generate(
3 input_ids,
4 images_or_videos=tensor,
5 modal_list=modal_list,
6 do_sample=True,
7 temperature=0.2,
8 max_new_tokens=1024,
9 use_cache=True,
10 stopping_criteria=[stopping_criteria]
11 )
13 outputs = tokenizer.batch_decode(output_ids, skip_special_tokens=True)
14 print(outputs[0])

File ~/anaconda3/lib/python3.11/site-packages/torch/utils/_contextlib.py:115, in context_decorator..decorate_context(*args, **kwargs)
112 @functools.wraps(func)
113 def decorate_context(*args, **kwargs):
114 with ctx_factory():
--> 115 return func(*args, **kwargs)

File /Dstorage/workspace/VideoLLaMA2/videollama2/model/language_model/videollama2_mistral.py:132, in Videollama2MistralForCausalLM.generate(self, inputs, images_or_videos, modal_list, **kwargs)
129 else:
130 inputs_embeds = self.get_model().embed_tokens(inputs)
--> 132 return super().generate(
133 position_ids=position_ids,
134 attention_mask=attention_mask,
135 inputs_embeds=inputs_embeds,
136 **kwargs
137 )

File ~/anaconda3/lib/python3.11/site-packages/torch/utils/_contextlib.py:115, in context_decorator..decorate_context(*args, **kwargs)
112 @functools.wraps(func)
113 def decorate_context(*args, **kwargs):
114 with ctx_factory():
--> 115 return func(*args, **kwargs)

File ~/anaconda3/lib/python3.11/site-packages/transformers/generation/utils.py:1525, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)
1517 input_ids, model_kwargs = self._expand_inputs_for_generation(
1518 input_ids=input_ids,
1519 expand_size=generation_config.num_return_sequences,
1520 is_encoder_decoder=self.config.is_encoder_decoder,
1521 **model_kwargs,
1522 )
1524 # 13. run sample
-> 1525 return self.sample(
1526 input_ids,
1527 logits_processor=prepared_logits_processor,
1528 logits_warper=logits_warper,
1529 stopping_criteria=prepared_stopping_criteria,
1530 pad_token_id=generation_config.pad_token_id,
1531 eos_token_id=generation_config.eos_token_id,
1532 output_scores=generation_config.output_scores,
1533 return_dict_in_generate=generation_config.return_dict_in_generate,
1534 synced_gpus=synced_gpus,
1535 streamer=streamer,
1536 **model_kwargs,
1537 )
1539 elif generation_mode == GenerationMode.BEAM_SEARCH:
1540 # 11. prepare beam search scorer
1541 beam_scorer = BeamSearchScorer(
1542 batch_size=batch_size,
1543 num_beams=generation_config.num_beams,
(...)
1548 max_length=generation_config.max_length,
1549 )

File ~/anaconda3/lib/python3.11/site-packages/transformers/generation/utils.py:2658, in GenerationMixin.sample(self, input_ids, logits_processor, stopping_criteria, logits_warper, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, synced_gpus, streamer, **model_kwargs)
2656 # sample
2657 probs = nn.functional.softmax(next_token_scores, dim=-1)
-> 2658 next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
2660 # finished sentences should have their next token be a padding token
2661 if eos_token_id is not None:

RuntimeError: probability tensor contains either `inf`, `nan` or element < 0"
}

Start with videollama2/model/language_model/videollama2_mistral.py at line 132 and reproduce the model.generate call from the demo using the reported model, video, and generation settings. Read the surrounding generate path and the Transformers sampling traceback; done means identifying a reproducible cause and confirming generation produces valid output without the probability-tensor error.

Written by the indexing model from the issue text.