facebookresearch / facebookresearch/deepconf

Majority Vote and Weighted majority vote does not group by equivalent answers?

Open
#15 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
412
Forks
62
PR merge metrics
No merged PRs in 30d

Description

Hi!

Thanks for open sourcing the implementation of DeepThink. I was looking into the implementation of `simple_majority_vote` and `weighted_majority_vote` [here](https://github.com/anirudhb11/deepconf/blob/main/deepconf/utils.py#L67-L90), and one thing I noticed is that the implementation does not group equivalent answers together. So for a given question, if the model answers `0.5` or `\frac{1}{2}`, these will be counted as two separate answers, when in reality they are the same answer.

Is this intended?

Thanks!

Contributor guide

Open the contributing guide

Research direction

Start in deepconf/utils.py at lines 67-90 and inspect simple_majority_vote and weighted_majority_vote. Reproduce the reported case with answers such as 0.5 and \frac{1}{2}; done means equivalent answer representations are grouped consistently by both voting functions, with the behavior covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.