Issues with Running video_chat2 on Multi-GPU Setup with Nvidia Titan Xp

Hi,

I'm currently attempting to run the video_chat2 model on a multi-GPU setup consisting of 8 Nvidia Titan Xp GPUs, each with 12GiB of memory. I'm using the mvbench.ipynb notebook from the Ask-Anything/video_chat2 repository for this purpose.

To ensure the model loads on my GPUs, I've enabled the low_resource option in config.json. Additionally, I've specified device_map="auto" during the initialization of the llama_model in videochat2_it.py. The relevant code snippet is as follows:

if self.low_resource:
    self.llama_model = LlamaForCausalLM.from_pretrained(
        llama_model_path,
        load_in_8bit=True,
        device_map="auto",
        torch_dtype=torch.float16,
    )

However, when I execute the code, I encounter multiple errors originating from the following lines:

seg_embs = [model.llama_model.base_model.model.model.embed_tokens(seg_t).cpu() for seg_t in seg_tokens] # get_context_emb

outputs = model.llama_model.generate()

Could you provide some guidance or suggestions on how to effectively perform inference with sharded models in this multi-GPU environment?

Thank you for your incredible works.

Start with mvbench.ipynb and the llama_model initialization in videochat2_it.py, then inspect the reported failures at the seg_embs and generate calls under device_map="auto" and low_resource. Done means establishing whether inference works with the sharded model on the eight Titan Xp GPUs and documenting the supported approach.