InternLM / InternLM/lmdeploy

[Bug] Identical inputs produce different results with temperature=0

Open
#3,963 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

I deployed my fine-tuned model using lmdeploy and connected it to the OpenAI API through a gRPC proxy. Recently, I noticed that for the same request and the same image, the model sometimes returns different results, even though I have already confirmed that temperature is set to 0. Later, I also set both top_k and top_p to 1, but the outputs can still vary.

Why is this happening?
Is there any way to explicitly control parameters such as do_sample (to disable sampling entirely) so that the model becomes strictly deterministic?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.