lm-sys / lm-sys/FastChat

Add Undo Feature to Model Evaluation Process

Open
#2,887 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

**Summary:**
As a user of FastChat, I've encountered a scenario where I've accidentally evaluated a model with no option to revert or cancel the action. This can be frustrating and potentially lead to wasted resources. I propose an enhancement to introduce an undo functionality for the model evaluation process.

**Problem:**
There is no current method to cancel or undo a model evaluation once the 'Evaluate' button has been clicked.

**Proposed Solution:**
Implement an undo option that provides a short time window (e.g., 5-10 seconds) after initiating a model evaluation during which a user can cancel the action. Only then the models could be displayed.

**Acceptance Criteria:**
- After clicking 'Evaluate', an 'Undo' button should appear for a predefined time window (suggestion: 5-10 seconds).
- If the 'Undo' button is clicked within this time frame, the evaluation process is stopped, and no data is recorded.
- If the time window elapses without the 'Undo' button being clicked, the model evaluation proceeds as normal.

**Potential Benefits:**
- Reduces the risk of accidental model evaluations.
- Provides users with a sense of control and flexibility.
- Improves user experience by preventing unnecessary frustrations.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by locating the model evaluation UI and the existing Evaluate action, then determine how cancellation and recording are handled; done means an Undo control appears for the chosen window, stops evaluation without recording data, and otherwise preserves normal evaluation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
frontend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.