Cannot succesfully pull VLMs to fastchat

Nobody has claimed this yet.

For anyone that has gotten VLMs to work in fastchat. How did you do so? I cannot even pull any llava model from hugging face successfully. These have been my results so far:
```
python -m fastchat.serve.cli --model-path llava-hf/llava-v1.6-vicuna-13b-hf
/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
tokenizer_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.18k/1.18k [00:00<00:00, 249kB/s]
tokenizer.model: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500k/500k [00:00<00:00, 6.64MB/s]
added_tokens.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 23.0/23.0 [00:00<00:00, 21.8kB/s]
special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 552/552 [00:00<00:00, 366kB/s]
tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.84M/1.84M [00:00<00:00, 7.02MB/s]
config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.34k/1.34k [00:00<00:00, 284kB/s]
Traceback (most recent call last):
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/fastchat/serve/cli.py", line 304, in
main(args)
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/fastchat/serve/cli.py", line 227, in main
chat_loop(
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/fastchat/serve/inference.py", line 361, in chat_loop
model, tokenizer = load_model(
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/fastchat/model/model_adapter.py", line 353, in load_model
model, tokenizer = adapter.load_model(model_path, kwargs)
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/fastchat/model/model_adapter.py", line 689, in load_model
model = AutoModelForCausalLM.from_pretrained(
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
raise ValueError(
ValueError: Unrecognized configuration class for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, LlamaConfig, CodeGenConfig, CohereConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, DbrxConfig, ElectraConfig, ErnieConfig, FalconConfig, FuyuConfig, GemmaConfig, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, JambaConfig, LlamaConfig, MambaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MistralConfig, MixtralConfig, MptConfig, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, OlmoConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PersimmonConfig, PhiConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, Qwen2Config, Qwen2MoeConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, Speech2Text2Config, StableLmConfig, Starcoder2Config, TransfoXLConfig, TrOCRConfig, WhisperConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig.
```

```
python -m fastchat.serve.cli --model-path llava-hf/llava-1.5-13b-hf
Traceback (most recent call last):
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/fastchat/serve/cli.py", line 304, in
main(args)
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/fastchat/serve/cli.py", line 227, in main
chat_loop(
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/fastchat/serve/inference.py", line 361, in chat_loop
model, tokenizer = load_model(
File "/home/philip/miniconda3/envs/myfastchat/lib/python3.9/site-packages/fastchat/model/model_adapter.py", line 353, in load_model
model, tokenizer = adapter.load_model(model_path, kwa
```

I am using a Conda Environment (Python 3.9.19) if that is relevant.

Start by reproducing the two commands in fastchat/serve/cli.py and trace model loading through fastchat/serve/inference.py and fastchat/model/model_adapter.py. Compare the reported LlavaNextConfig failure with the AutoModelForCausalLM loading path; done means the named Llava models can be loaded through the CLI or the supported limitation is documented.

Written by the indexing model from the issue text.