facebookresearch / facebookresearch/perception_models

Region Captioning Eval on PLM-VideoBench

Open
#95 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.4k
Forks
162
PR merge metrics
No merged PRs in 30d

Description

Hello,
I have checked Appendix C.3 for the evaluation protocol on PLM-VideoBench. The prompt you use to Llama to evaluate the region caption is written as the following:
"Your task is to compare a given pair of captions and provide a single score indicating how correct the pred is compared to GT, on a scale from 0 to 10. Focus on meaning and context, not exact word matches. Penalize missing and incorrect information, with lower scores for more significant errors. High scores require accurate conveyance of all key GT information. Respond with only the score, starting your response with the number and including no additional text. Output format: [score]."
However, there is no way the LLM would know which is the groundtruth and which is the prediction from the input. How can it identify the missing information. It seems you prompted with the output format but not the input format? Am I missing something?

Also can you explain how the aggregation of the metric occurs with the 1-10 scores to match the numbers in Table 5 at 46.6 are you summing the scores across the examples? Cause if you are averaging it should have been a number from 1-10.

Thank you,

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.