AnswerDotAI / AnswerDotAI/cold-compress
Question of evaluation
- Dominant language
- Python
- Stars
- 153
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
Thank you for providing the code to easily test various KV-related algorithms.
I have a question regarding evaluation.
I compared evaluations through truthfulQA.
Accuracy was recorded in "truthfulqa_metrics.json". When the compression ratio was set to about 90% using H2O, the accuracy was about 0.57, and when the compression was set to 10%, the accuracy decreased to about 0.43.

It seems that the prediction results are recorded in the truthfulqa_predictions.csv file, but when I opened the excel file, it was confirmed that the correct answer was not written in the prediction.
I would like to ask for guidance on whether it is working properly.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.