facebookresearch / facebookresearch/CRAG
How to use task #1 dataset for our own evaluation?
- Dominant language
- Jupyter Notebook
- Stars
- 303
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
Hi guys, first of all, congrats for the beautiful dataset!! 👏 👏
I have a question about the dataset for tasks 1 and 2 (`crag_task_1_and_2_dev_v4.jsonl.bz2`): we are currently using the dataset to evaluate our RAG system, using our own evaluation pipeline. Now, for task 1 and 2 "5 pages are randomly selected from the top-10 pages. These pages are likely relevant to the question, but relevance is not guaranteed.".
The question is then: how can we use the gold-standard answer for the `crag_task_1_and_2_dev_v4.jsonl.bz2` dataset, as we don't know if the 5 randomly selected pages contain the right context from which the gold-standard answer has been obtained?
Perhaps I misunderstood something, but since the url of the web page used to generate the gold-standard answer is not included in the `crag_task_1_and_2_dev_v4.jsonl.bz2` file, we cannot infer whether a question can/cannot be correctly answered using the 5 web pages.
Thanks for any hint!
Contributor guide
Assessment
This issue has not been assessed yet.