huggingface / huggingface/lighteval

[BUG] Model Parallelism is incorrectly disabled on single multi-GPU machines

Open
#895 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
When running an evaluation with accelerate launch on a single machine with multiple GPUs, setting model_parallel=True in the TransformersModelConfig has no effect. The library logs that it is "not in a distributed setting" and force-overrides model_parallel to False.

## To Reproduce
Set up an evaluation script on a single machine with 2 or more GPUs.
In the script, create a TransformersModelConfig for a large model (e.g., 7B+) and explicitly set model_parallel=True.
Launch the script using accelerate launch --num_processes=2 your_script.py.
Observe: We are not in a distributed setting. Setting model_parallel to False.

## Expected behavior
lighteval should respect the model_parallel=True configuration in a single-node, multi-GPU environment. It should proceed to shard the model across the available GPUs using device_map="auto" instead of disabling the feature.

## Version info
build from source code

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.