The current `device_map` had weights offloaded to the disk. Please provide an `offload_folder` for them. Alternatively, make sure you have `safetensors` installed if the model you are using offers the weights in this format.

Nobody has claimed this yet.

An error occurs when vicuna-13B-1.1 is executed

Here is the video card in my environment

Scenario 1: There are 2 graphics cards with 32GB memory, and 26GB is sufficient for running the 13B model. However, due to the line of code in model_adapter.py at line 112 that sets the GPU memory usage to 85%, which results in only about 25GB being available, causing an out-of-memory error.

Scenario 2: Using the --max-gpu-memory 20GiB flag should work, but the program loads 12.5GB of memory on the first card and the remaining on the second card, resulting in an error. From line 116 of model_adapter.py, we can see that the maximum memory usage is set to 20GB for both cards, and the program assumes it is enough to split the resources equally.

Solution:
Manually modify the threshold value in line 112 of the code to be 0.9.
Modify line 116 of the code and add logic to calculate the actual GPU memory size based on available_gpu_memory.

Scenario 3: After modifying line 116 of the code in accordance with Solution 2, an error occurs when kwargs["device_map"] = "auto" because the kwargs["max_memory"] values are the same. The full error message is: "The current device_map had weights offloaded to the disk. Please provide an offload_folder for them. Alternatively, make sure you have safetensors installed if the model you are using offers the weights in this format".

Modified code:

Memory usage after modification:

Reproduce the vicuna-13B-1.1 loading scenarios, then inspect model_adapter.py around lines 112 and 116, including handling of --max-gpu-memory and available GPU memory. Compare the resulting device_map and max_memory values across both GPUs. Done means the model loads without an out-of-memory or disk-offload error under the reported configurations.

Written by the indexing model from the issue text.