[Bug] Identical inputs produce different results with temperature=0
Open
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
I deployed my fine-tuned model using lmdeploy and connected it to the OpenAI API through a gRPC proxy. Recently, I noticed that for the same request and the same image, the model sometimes returns different results, even though I have already confirmed that temperature is set to 0. Later, I also set both top_k and top_p to 1, but the outputs can still vary.
Why is this happening?
Is there any way to explicitly control parameters such as do_sample (to disable sampling entirely) so that the model becomes strictly deterministic?
Contributor guide
Assessment
This issue has not been assessed yet.