Error Running cogvlm model on Self-Hosted GPU Server with Roboflow Inference (Transformer Version)

Search before asking

• I have searched the Inference issues and found no similar bug report.

Bug

I'm encountering an issue while attempting to deploy the cogvlm model on my own GPU server using Roboflow inference code. The server setup seems to be correct, but when I try to run the model, I run into the following error:

  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3502, in from_pretrained
    ):
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3926, in _load_pretrained_model
    model._is_quantized_training_enabled = True
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 802, in _load_state_dict_into_meta_model
    state_dict_index = offload_weight(param, param_name, state_dict_folder, state_dict_index)
  File "/usr/local/lib/python3.10/dist-packages/transformers/quantizers/quantizer_bnb_4bit.py", line 124, in check_quantized_param
KeyError: 'inv_freq'
INFO:     172.17.0.1:45116 - "POST /llm/cogvlm HTTP/1.1" 500 Internal Server Error

Upon further investigation and based on this GitHub issue (https://github.com/THUDM/CogVLM/issues/396), it's recommended to downgrade the transformers library to version 4.37 due to compatibility issues. However, the current deployment is using version 4.38. Could you please confirm if the transformers version could be the source of this issue and if downgrading would be appropriate? Any other insights or suggestions would also be greatly appreciated.
Thank you!

Environment

inference 0.9.20
inference-cli 0.9.20
inference-gpu 0.9.20
inference-sdk 0.9.20

x86-gpu(rtx3090)

Minimal Reproducible Example

cog-vlm-client$ python script.py --image "data/tire.jpg" --prompt "read serial number from tire"

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

Start by reproducing the request from script.py against the /llm/cogvlm endpoint and inspect how the deployed environment selects Transformers 4.38. Compare the CogVLM loading failure with the reported 4.37 compatibility constraint; done means the model loads successfully and the minimal request no longer returns HTTP 500.