huggingface / huggingface/lighteval
[BUG] Automatic batch size detection causes distributed deadlock in model parallel mode
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
When model parallelism is enabled and `batch_size` is not explicitly set in the config, the evaluation process hangs indefinitely. The log stops after the message "Detecting largest batch size...", indicating a distributed deadlock among the processes.
## To Reproduce
1. Configure an evaluation for a large, sharded model with model_parallel=True on a multi-GPU machine.
2. Crucially, do not specify a batch_size in the `TransformersModelConfig`, to allow the automatic detection feature to trigger.
3. Launch the script using `accelerate launch --num_processes > 1`.
4. Observe that the script and all worker processes hang permanently with no further output after the batch size detection log message appears.
## Expected behavior
The automatic batch size detection should either be compatible with model parallelism or be automatically skipped to prevent deadlocks. The evaluation should proceed with a safe default batch size (e.g., 1) or raise an error prompting the user to set a manual batch size when in model parallel mode.
## Version info
build from source code
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.