Error: Llama-2-7B-Chat-AWQ safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

Nobody has claimed this yet.

I'm trying to host the `TheBloke/Llama-2-7B-Chat-AWQ` model with fastchat on GCP's N1 machine (has a 15GB memory GPU) with this commend:
```bash
python3 -m fastchat.serve.controller --host 0.0.0.0 &
python3 -m fastchat.serve.vllm_worker --host 0.0.0.0 --model-path saved_models/Llama-2-7B-Chat-AWQ --quantization awq --num-gpus 1
```
and got this error:
```
WARNING 01-31 01:16:58 config.py:175] awq quantization is not fully optimized yet. The speed can be slower than non-quantized models.
INFO 01-31 01:16:58 llm_engine.py:70] Initializing an LLM engine with config: model='saved_models/Llama-2-7B-Chat-AWQ', tokenizer='saved_models/Llama-2-7B-Chat-AWQ', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=awq, enforce_eager=False, seed=0)
2024-01-31 01:17:03 | ERROR | stderr | Traceback (most recent call last):
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
2024-01-31 01:17:03 | ERROR | stderr | return _run_code(code, main_globals, None,
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
2024-01-31 01:17:03 | ERROR | stderr | exec(code, run_globals)
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/fastchat/serve/vllm_worker.py", line 271, in
2024-01-31 01:17:03 | ERROR | stderr | engine = AsyncLLMEngine.from_engine_args(engine_args)
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 500, in from_engine_args
2024-01-31 01:17:03 | ERROR | stderr | engine = cls(parallel_config.worker_use_ray,
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 273, in __init__
2024-01-31 01:17:03 | ERROR | stderr | self.engine = self._init_engine(*args, **kwargs)
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 318, in _init_engine
2024-01-31 01:17:03 | ERROR | stderr | return engine_class(*args, **kwargs)
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 111, in __init__
2024-01-31 01:17:03 | ERROR | stderr | self._init_workers()
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 146, in _init_workers
2024-01-31 01:17:03 | ERROR | stderr | self._run_workers("load_model")
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 795, in _run_workers
2024-01-31 01:17:03 | ERROR | stderr | driver_worker_output = getattr(self.driver_worker,
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 81, in load_model
2024-01-31 01:17:03 | ERROR | stderr | self.model_runner.load_model()
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 64, in load_model
2024-01-31 01:17:03 | ERROR | stderr | self.model = get_model(self.model_config)
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/model_loader.py", line 72, in get_model
2024-01-31 01:17:03 | ERROR | stderr | model.load_weights(model_config.model, model_config.download_dir,
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 313, in load_weights
2024-01-31 01:17:03 | ERROR | stderr | for name, loaded_weight in hf_model_weights_iterator(
2024-01-31 01:17:03 | ERROR | stderr | File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/weight_utils.py", line 239, in hf_model_weights_iterator
2024-01-31 01:17:03 | ERROR | stderr | with safe_open(st_file, framework="pt") as f:
2024-01-31 01:17:03 | ERROR | stderr | safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
```

Anyone has any idea what's going wrong? Is it a memory issue or something different?
Thanks!

Start by reproducing the shown vllm_worker command on the GCP setup and follow the traceback from fastchat/serve/vllm_worker.py into vllm's model loading path. Inspect the saved_models/Llama-2-7B-Chat-AWQ files involved in the safetensors_rust HeaderTooLarge failure. Done means identifying and documenting a reproducible cause or a confirmed compatibility problem.

Written by the indexing model from the issue text.