huggingface / huggingface/lighteval

[BUG] Automatic batch size detection causes distributed deadlock in model parallel mode

Open
#896 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
When model parallelism is enabled and `batch_size` is not explicitly set in the config, the evaluation process hangs indefinitely. The log stops after the message "Detecting largest batch size...", indicating a distributed deadlock among the processes.

## To Reproduce
1. Configure an evaluation for a large, sharded model with model_parallel=True on a multi-GPU machine.
2. Crucially, do not specify a batch_size in the `TransformersModelConfig`, to allow the automatic detection feature to trigger.
3. Launch the script using `accelerate launch --num_processes > 1`.
4. Observe that the script and all worker processes hang permanently with no further output after the batch size detection log message appears.

## Expected behavior
The automatic batch size detection should either be compatible with model parallelism or be automatically skipped to prevent deadlocks. The evaluation should proceed with a safe default batch size (e.g., 1) or raise an error prompting the user to set a manual batch size when in model parallel mode.
## Version info
build from source code

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.