deepspeedai / deepspeedai/DeepSpeed

[REQUEST] Allow eval batch size in pipeline

Open
#3,045 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

When I use PipelineModule.eval_batch(eval_iter) in my code, I found it necessary to set eval dataloader's batch_size same as train_micro_batch_size_per_gpu. For example, I set train_micro_batch_size_per_gpu to 2, and set eval dataloader' batch_size to 8:

val_dataloader = RepeatingLoader(DataLoader(val_dataset, 8))
eval_iter = iter(val_dataloader)

Then run train_batch and eval_batch:

for i in range(epochs):
    model.train_batch()
    model.eval_batch(eval_iter)

Then at the end of the first stage, there is an output tensor e.g. hidden_state, whose shape is 2, 16, 1024. But at the second stage, its shape changed to 8, 16, 1024, which may cause some unexpected results.

reset_activation_shape can solve it, but in my opinion calling reset_activation_shape before and after evaluation is a bit weird...so I think it may be more convenient to support this feature. Sorry to bother :)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing PipelineModule.eval_batch and the existing reset_activation_shape behavior, comparing them with train_batch and the eval_iter example in the issue. The change is complete when evaluation can use a different dataloader batch size without callers manually resetting activation shape, while outputs remain consistent across pipeline stages.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.