[Bug] Single quote character `'` tripping up model generation and streaming

I am using FastChat to serve a Mixtral 8x7B model using the OpenAI-Compatible RESTful API. However, I'm seeing some inconsistent generation results, specifically with the single quote ' character.

The issue is that the stream generation seems to somehow eat the first character after the single quote sometimes. This is what the streaming result on the client side ends up looking like:

Prompt:
Think about the word 'sanitize'

Stream:
Thought
Thought: The
Thought: The user has
Thought: The user has asked me
Thought: The user has asked me to think
Thought: The user has asked me to think about the
Thought: The user has asked me to think about the word '
Thought: The user has asked me to think about the word 'anitize

Looking into chat_completion_stream_generator() inside openai_api_server.py, content["text"] ends up looking like so:

<snip>
Thought: The user has asked me to think about the
Thought: The user has asked me to think about the word '
Thought: The user has asked me to think about the word'sanitize

Note the space before the single quote disappearing. This, when combined with the logic to generate a new chunk by removing characters equal to the length of previous text (decoded_unicode[len(previous_text) :]), ends up eating up the s character after the single quote.

So I started to look into why the space was disappearing and I ended this code in generate_stream() in inference.py:

output = tokenizer.decode(
    tmp_output_ids,
    skip_special_tokens=True,
    spaces_between_special_tokens=False,
    clean_up_tokenization_spaces=True,
)

Here, the clean_up_tokenization_spaces argument ends up stripping the space before the single quote.

To verify, I conducted a test with the Mixtral tokenizer

import torch
from transformers import AutoTokenizer

device = "auto"
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1", device_map=device)

initial_token_ids = [26142, 28747, 415, 2188, 659, 2261, 528, 298, 1073, 684, 272, 1707, 464]
next_token_ids = [20297, 23175]

kwargs = {
    'skip_special_tokens': True,
    'spaces_between_special_tokens': False,
    'clean_up_tokenization_spaces': False,
}

print("Initial tokens, clean_up_tokenization_spaces=False")
print('"' + tokenizer.decode(
    initial_token_ids,
    **kwargs
) + '"')
print()

print("Initial + next tokens, clean_up_tokenization_spaces=False")
print('"' + tokenizer.decode(
    initial_token_ids + next_token_ids,
    **kwargs
) + '"')
print()

kwargs['clean_up_tokenization_spaces'] = True

print("Initial tokens, clean_up_tokenization_spaces=True")
print('"' + tokenizer.decode(
    initial_token_ids,
    **kwargs
) + '"')
print()

print("Initial + next tokens, clean_up_tokenization_spaces=True")
print('"' + tokenizer.decode(
    initial_token_ids + next_token_ids,
    **kwargs
) + '"')
print()

which gives me this result:

Initial tokens, clean_up_tokenization_spaces=False
"Thought: The user has asked me to think about the word '"

Initial + next tokens, clean_up_tokenization_spaces=False
"Thought: The user has asked me to think about the word 'sanitize"

Initial tokens, clean_up_tokenization_spaces=True
"Thought: The user has asked me to think about the word '"

Initial + next tokens, clean_up_tokenization_spaces=True
"Thought: The user has asked me to think about the word'sanitize"

Start in inference.py at generate_stream() and then inspect chat_completion_stream_generator() in openai_api_server.py. Reproduce the Mixtral tokenizer example from the issue and verify that streaming preserves the space before a quoted word and does not drop the following character; add or update coverage for this case if the project’s existing tests identify a suitable location.