AnswerDotAI / AnswerDotAI/cold-compress

Question of evaluation

Open
#43 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
153
Forks
16
PR merge metrics
No merged PRs in 30d

Description

Thank you for providing the code to easily test various KV-related algorithms.

I have a question regarding evaluation.

I compared evaluations through truthfulQA.
Accuracy was recorded in "truthfulqa_metrics.json". When the compression ratio was set to about 90% using H2O, the accuracy was about 0.57, and when the compression was set to 10%, the accuracy decreased to about 0.43.
![image](https://github.com/user-attachments/assets/c71a51bc-279c-41f5-8a95-2dd40ceac59e)

It seems that the prediction results are recorded in the truthfulqa_predictions.csv file, but when I opened the excel file, it was confirmed that the correct answer was not written in the prediction.

I would like to ask for guidance on whether it is working properly.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.