facebookresearch / facebookresearch/dpr-scale

DRAMA Model Demo output not matching

Open
#20 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
297
Forks
33
PR merge metrics
No merged PRs in 30d

Description

Hi Team, I am trying to test the DRAMA model sample code - [link](https://github.com/facebookresearch/dpr-scale/tree/main/drama). However, I see mismatch between the expected output and actual output.

Sample code:
```import torch
from transformers import AutoTokenizer, AutoModel

queries = [
'What percentage of the Earth\'s atmosphere is oxygen?',
'意大利首都是哪里?',
]
documents = [
"The amount of oxygen in the atmosphere has fluctuated over the last 600 million years, reaching a peak of 35% during the Carboniferous period, significantly higher than today's 21%.",
"羅馬是欧洲国家意大利首都和罗马首都广域市的首府及意大利全国的政治、经济、文化和交通中心,位于意大利半島中部的台伯河下游平原地,建城初期在七座小山丘上,故又名“七丘之城”。按城市范围内的人口计算,罗马是意大利人口最多的城市,也是欧盟人口第三多的城市。",
]

model_name = "facebook/drama-base"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).to(device)

query_embs = model.encode_queries(tokenizer, queries)
doc_embs = model.encode_documents(tokenizer, documents)

scores = query_embs @ doc_embs.T
print(scores.tolist())
Expected output: [[0.5310, 0.0821], [0.1298, 0.6181]]
```

Actual output - [[0.4584735929965973, 0.24322254955768585], [0.12728893756866455, 0.5092089176177979]]

Colab Notebook link - https://colab.research.google.com/drive/1FkJMGEJBX7BGsoLeGiJdxCKnMBmMG19n?usp=sharing

What's causing this issue? I also tested the sample code with new values
queries = [
'iphone', 'cat food'
]

documents = [
'iphone 16 pro max',
'best cat food'
]

output - [[0.40802454948425293, 0.26841771602630615], [0.27385222911834717, 0.5687180757522583]]

Is this the correct behavior? The relevance seems quite poor

Thanks

Contributor guide

Open the contributing guide

Research direction

Start with the DRAMA sample code in the issue and compare its expected and actual scores, then reproduce the run using the linked Colab notebook. Check the loaded model and runtime assumptions before investigating the score differences. Done means identifying the cause of the mismatch or confirming the correct output and explaining the relevance results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.