huggingface / huggingface/chat-ui

top_k and frequency_penalty not passed to OpenAI compatible API endpoint correctly

Open
#895 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
11k
Forks
1.7k
Avg merge
21h 24m
Merged PRs (30d)
56

Description

I deployed chat-ui as a frontend for dolphin-2.6-mixtral-8x7b, CodeFuse-DeepSeek-33B GPTQ models etc.
In my case text generation webui serves the model inference. I found that theses models works well in the chat page on text generation webui. But gives worse response in chat-ui front end.

I checked the verbose log in text generation webui. And finally confirmed the problems was caused by top_k and frequency_penalty not passed. I update the code in src/lib/server/endpoints/openai/endpointOai.ts, finally it works with OpenAI API exposed by text-generation-webui.

```diff
@@ -55,7 +55,8 @@ export async function endpointOai(
stop: model.parameters?.stop,
temperature: model.parameters?.temperature,
top_p: model.parameters?.top_p,
- frequency_penalty: model.parameters?.repetition_penalty,
+ top_k: 21,
+ frequency_penalty: 0,
})
);
};
@@ -79,7 +80,8 @@ export async function endpointOai(
stop: model.parameters?.stop,
temperature: model.parameters?.temperature,
top_p: model.parameters?.top_p,
- frequency_penalty: model.parameters?.repetition_penalty,
+ frequency_penalty: 0,
+ top_k: 21,
})
);
};
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/lib/server/endpoints/openai/endpointOai.ts and inspect both request-building paths shown in the issue. Compare the parameters sent to the text-generation-webui OpenAI-compatible endpoint with its verbose log; done means top_k and frequency_penalty are passed correctly and the affected models produce the expected responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.