Delayed Response
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
When comparing a reasoning model with a non-reasoning model, disable streaming, and post both responses to the user at the same time after both of the models are done to prevent the user from using model response times (ie longer response time = some sort of reasoning model) to bias voting.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing FastChat's model-comparison response handling and streaming behavior. Confirm that streaming is disabled for these comparisons and that both completed responses are posted together, so response duration does not reveal which model uses reasoning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100