huggingface / huggingface/chat-ui
top_k and frequency_penalty not passed to OpenAI compatible API endpoint correctly
- Dominant language
- TypeScript
- Stars
- 11k
- Forks
- 1.7k
- Avg merge
- 21h 24m
- Merged PRs (30d)
- 56
Description
I deployed chat-ui as a frontend for dolphin-2.6-mixtral-8x7b, CodeFuse-DeepSeek-33B GPTQ models etc.
In my case text generation webui serves the model inference. I found that theses models works well in the chat page on text generation webui. But gives worse response in chat-ui front end.
I checked the verbose log in text generation webui. And finally confirmed the problems was caused by top_k and frequency_penalty not passed. I update the code in src/lib/server/endpoints/openai/endpointOai.ts, finally it works with OpenAI API exposed by text-generation-webui.
```diff
@@ -55,7 +55,8 @@ export async function endpointOai(
stop: model.parameters?.stop,
temperature: model.parameters?.temperature,
top_p: model.parameters?.top_p,
- frequency_penalty: model.parameters?.repetition_penalty,
+ top_k: 21,
+ frequency_penalty: 0,
})
);
};
@@ -79,7 +80,8 @@ export async function endpointOai(
stop: model.parameters?.stop,
temperature: model.parameters?.temperature,
top_p: model.parameters?.top_p,
- frequency_penalty: model.parameters?.repetition_penalty,
+ frequency_penalty: 0,
+ top_k: 21,
})
);
};
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/lib/server/endpoints/openai/endpointOai.ts and inspect both request-building paths shown in the issue. Compare the parameters sent to the text-generation-webui OpenAI-compatible endpoint with its verbose log; done means top_k and frequency_penalty are passed correctly and the affected models produce the expected responses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100