lm-sys / lm-sys/FastChat

better way of processing the Content Warning

Open
#3,189 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

When a Content Warning is triggered by user input, the warning message is still being sent to the language model and a response is generated, instead of the conversation stopping.

**Steps to Reproduce**
1. Go to chat.lmsys.org
2. Enter a message that should trigger a Content Warning (e.g. explicit language, violence, etc.)
3. The Content Warning is displayed, and sent to the language model
4. A response to the Content Warning is returned

**Expected Behavior**
When a Content Warning is triggered, the conversation should stop and no further processing or response generation should occur from the language model.

**Actual Behavior**
The Content Warning is displayed, but the content warning message is still sent to the language model, which produces a response. This wastes resources and may provide no additional insight.

**Severity**
Medium - This is a functional bug that goes against expected behavior for Content Warnings. However, it does not expose any sensitive data.

**Suggested Fix**
Implement logic to stop processing the user's message as soon as a Content Warning is triggered. The conversation flow should be halted at that point, without sending the message to the language model.

**Additional Notes**
Continuing to process messages that trigger Content Warnings is inefficient and provides no value to the end user. Addressing this bug would improve the user experience and conserve resources.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file, test, or entry point is named; start by tracing the chat flow that handles Content Warnings and the subsequent language-model request. Reproduce the reported case, then verify that a triggered warning stops the conversation before any response is generated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.