Issues Loading NF4 Model with --mode=2 --zh

Hi i am trying to make a 1 button installer for this model and i have manage to make Original model work and int8 model too but the NF4 quant is not working as there is no llm_config.json on [calcuis/bagel-gguf](https://huggingface.co/calcuis/bagel-gguf/tree/main) only the raw safe tensor files.

is there a example config for NF4 load that would work this is the following error using standard model config:

`The safetensors archive passed at D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\models\calcuis\bagel-gguf\ema.safetensors does not contain metadata. Make sure to save your model with the `save_pretrained` method. Defaulting to 'pt' metadata.
Traceback (most recent call last):
File "D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\app.py", line 116, in
model = load_and_quantize_model(
File "D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\venv\lib\site-packages\accelerate\utils\bnb.py", line 183, in load_and_quantize_model
load_checkpoint_in_model(
File "D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\venv\lib\site-packages\accelerate\utils\modeling.py", line 2015, in load_checkpoint_in_model
set_module_tensor_to_device(
File "D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\venv\lib\site-packages\accelerate\utils\modeling.py", line 292, in set_module_tensor_to_device
raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([76032, 3584]) in "weight" (which has shape torch.Size([152064, 3584])), this looks incorrect.
Failed to download model or startup app.py.`

changing the vocab_size to match the model "76032" i get a different error as i don't have any information on what most of the configs do i cant really see how to fix this here s the error when running with the vocab size :

`The safetensors archive passed at D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\models\calcuis\bagel-gguf\ema.safetensors does not contain metadata. Make sure to save your model with the `save_pretrained` method. Defaulting to 'pt' metadata.
Traceback (most recent call last):
File "D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\app.py", line 116, in
model = load_and_quantize_model(
File "D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\venv\lib\site-packages\accelerate\utils\bnb.py", line 183, in load_and_quantize_model
load_checkpoint_in_model(
File "D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\venv\lib\site-packages\accelerate\utils\modeling.py", line 2015, in load_checkpoint_in_model
set_module_tensor_to_device(
File "D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\venv\lib\site-packages\accelerate\utils\modeling.py", line 363, in set_module_tensor_to_device
new_value = param_cls(new_value, requires_grad=old_value.requires_grad).to(device)
File "D:\test_install\Bagel_1Click_V02\Bagel_1Click_V02\code\venv\lib\site-packages\torch\nn\parameter.py", line 46, in __new__
return torch.Tensor._make_subclass(cls, data, requires_grad)
RuntimeError: Only Tensors of floating point and complex dtype can require gradients`

this is the current llm_config.json :

`{
"vocab_size": 76032,
"max_position_embeddings": 32768,
"hidden_size": 3584,
"intermediate_size": 18944,
"num_hidden_layers": 28,
"num_attention_heads": 28,
"use_sliding_window": false,
"sliding_window": null,
"max_window_layers": 28,
"num_key_value_heads": 4,
"hidden_act": "silu",
"initializer_range": 0.02,
"rms_norm_eps": 1e-06,
"use_cache": true,
"rope_theta": 1000000.0,
"rope_scaling": null,
"attention_dropout": 0.0,
"is_causal": true,
"return_dict": true,
"output_hidden_states": false,
"output_attentions": false,
"torchscript": false,
"torch_dtype": "bfloat16",
"use_bfloat16": false,
"tf_legacy_loss": false,
"pruned_heads": {},
"tie_word_embeddings": false,
"chunk_size_feed_forward": 0,
"is_encoder_decoder": false,
"is_decoder": false,
"cross_attention_hidden_size": null,
"add_cross_attention": false,
"tie_encoder_decoder": false,
"max_length": 20,
"min_length": 0,
"do_sample": false,
"early_stopping": false,
"num_beams": 1,
"num_beam_groups": 1,
"diversity_penalty": 0.0,
"temperature": 1.0,
"top_k": 50,
"top_p": 1.0,
"typical_p": 1.0,
"repetition_penalty": 1.0,
"length_penalty": 1.0,
"no_repeat_ngram_size": 0,
"encoder_no_repeat_ngram_size": 0,
"bad_words_ids": null,
"num_return_sequences": 1,
"output_scores": false,
"return_dict_in_generate": false,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"remove_invalid_values": false,
"exponential_decay_length_penalty": null,
"suppress_tokens": null,
"begin_suppress_tokens": null,
"architectures": [
"Qwen2ForCausalLM"
],
"finetuning_task": null,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"tokenizer_class": null,
"prefix": null,
"bos_token_id": 151643,
"pad_token_id": null,
"eos_token_id": 151645,
"sep_token_id": null,
"decoder_start_token_id": null,
"task_specific_params": null,
"problem_type": null,
"_name_or_path": "",
"_attn_implementation_autoset": true,
"transformers_version": "4.49.0",
"model_type": "qwen2",
"qk_norm": true,
"layer_module": "Qwen2MoTDecoderLayer",
"freeze_und": false
}`

if some one has any clue or idea on how to get this to work please leave a comment ( the issue seams to be the missing metadata in the NF4 Model) and the value type (floating point and complex dtype) for gradient

"""edit"""
almost forgot :
specs
System: Windows 11
GPU RTX3090 24gb vram
CPU:RYZEN 5900X
RAM DDR4 3200 - 64GB

Start at app.py line 116 and the llm_config.json used by load_and_quantize_model; compare the NF4 checkpoint's tensor shapes and metadata with the Qwen2 configuration and the Accelerate/PyTorch loading errors. Done means NF4 loads successfully in the reported --mode=2 --zh setup without the shape or gradient-dtype errors.

Written by the indexing model from the issue text.