ByteDance-Seed / ByteDance-Seed/WideSearch

Potential bug in `evaluate_single_query`

Open
#7 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
154
Forks
18
PR merge metrics
No merged PRs in 30d

Description

If I understand it correctly, for the preprocess in function `evaluate_single_query`, we need to convert response and answer to the same type before comparing.
However, after checking Line 140-148, I notice that `answer_df` is converted to `str` first:
https://github.com/ByteDance-Seed/WideSearch/blob/0e664c307d12c88810a31530be40c8c358f56b4f/src/evaluation/evaluation.py#L140
But later it might be converted to float:
https://github.com/ByteDance-Seed/WideSearch/blob/0e664c307d12c88810a31530be40c8c358f56b4f/src/evaluation/evaluation.py#L147
while `response_df` is always converted to `str`:
https://github.com/ByteDance-Seed/WideSearch/blob/0e664c307d12c88810a31530be40c8c358f56b4f/src/evaluation/evaluation.py#L148

Is this a bug? If so, will it affect the final result greatly?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.