facebookresearch / facebookresearch/CRAG

How to use task #1 dataset for our own evaluation?

Open
#5 0 comments 4 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
303
Forks
39
PR merge metrics
No merged PRs in 30d

Description

Hi guys, first of all, congrats for the beautiful dataset!! 👏 👏

I have a question about the dataset for tasks 1 and 2 (`crag_task_1_and_2_dev_v4.jsonl.bz2`): we are currently using the dataset to evaluate our RAG system, using our own evaluation pipeline. Now, for task 1 and 2 "5 pages are randomly selected from the top-10 pages. These pages are likely relevant to the question, but relevance is not guaranteed.".

The question is then: how can we use the gold-standard answer for the `crag_task_1_and_2_dev_v4.jsonl.bz2` dataset, as we don't know if the 5 randomly selected pages contain the right context from which the gold-standard answer has been obtained?

Perhaps I misunderstood something, but since the url of the web page used to generate the gold-standard answer is not included in the `crag_task_1_and_2_dev_v4.jsonl.bz2` file, we cannot infer whether a question can/cannot be correctly answered using the 5 web pages.

Thanks for any hint!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.