AnswerDotAI / AnswerDotAI/byaldi

qwen2 models causing indexerror

Open
#80 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
851
Forks
91
PR merge metrics
No merged PRs in 30d

Description

I'm trying to run the example quick_overview.ipynb on m1 mbp.

```import os
from dotenv import load_dotenv
from pathlib import Path
from byaldi import RAGMultiModalModel

load_dotenv()
# print(os.environ["HF_TOKEN"])

# Initialize RAGMultiModalModel
model = RAGMultiModalModel.from_pretrained("vidore/colqwen2-v1.0", device="mps")

# Test indexing
metadata = [{"filename":file_name} for file_name in os.listdir("docs")]

index_name = "attention_index"
model.index(
input_path=Path("docs/"),
index_name=index_name,
store_collection_with_index=False,
metadata=metadata,
overwrite=True
)

# BLEU tables are on page 8 and 9. We've indexed the pdf and its evil mustached twin, so we should see similar scores occur twice for every relevant page.
query = "what's the BLEU score of this new strange method?"
results = model.search(query, k=5)

print(f"Search results for '{query}':")
for result in results:
print(f"Doc ID: {result.doc_id}, Page: {result.page_num}, Score: {result.score}")

print("Test completed successfully!")
```

But got the following error:

> ---------------------------------------------------------------------------
> IndexError Traceback (most recent call last)
> Cell In[6], [line 5](vscode-notebook-cell:?execution_count=6&line=5)
> [2](vscode-notebook-cell:?execution_count=6&line=2) metadata = [{"filename":file_name} for file_name in os.listdir("docs")]
> [4](vscode-notebook-cell:?execution_count=6&line=4) index_name = "attention_index"
> ----> [5](vscode-notebook-cell:?execution_count=6&line=5) model.index(
> [6](vscode-notebook-cell:?execution_count=6&line=6) input_path=Path("docs/"),
> [7](vscode-notebook-cell:?execution_count=6&line=7) index_name=index_name,
> [8](vscode-notebook-cell:?execution_count=6&line=8) store_collection_with_index=False,
> [9](vscode-notebook-cell:?execution_count=6&line=9) metadata=metadata,
> [10](vscode-notebook-cell:?execution_count=6&line=10) overwrite=True
> [11](vscode-notebook-cell:?execution_count=6&line=11) )
> [13](vscode-notebook-cell:?execution_count=6&line=13) # BLEU tables are on page 8 and 9. We've indexed the pdf and its evil mustached twin, so we should see similar scores occur twice for every relevant page.
> [14](vscode-notebook-cell:?execution_count=6&line=14) query = "what's the BLEU score of this new strange method?"
> ... ...
>
> File /opt/homebrew/Caskroom/miniforge/base/envs/colpali/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py:404, in VisionSdpaAttention.forward(self, hidden_states, cu_seqlens, rotary_pos_emb)
> [402](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/colpali/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py:402) k = k.transpose(0, 1)
> [403](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/colpali/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py:403) v = v.transpose(0, 1)
> --> [404](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/colpali/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py:404) attn_output = F.scaled_dot_product_attention(q, k, v, attention_mask, dropout_p=0.0)
> [405](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/colpali/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py:405) attn_output = attn_output.transpose(0, 1)
> [406](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/colpali/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py:406) attn_output = attn_output.reshape(seq_length, -1)
>
> IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)

It seems the colpali models works fine, but qwen2 models cause the same error. Thanks for any help how to fix this.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.