Lightning-AI / Lightning-AI/pytorch-lightning

Interleaved Model for combined_loader

Open
#17,910 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

data data handling feature
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

Description: A mode for combined_loader which would evenly distribute samples from all iterables.

Motivation: While working with cycle consistency training, two models are updated iteratively on the output of the other model. The current modes for combined_loader do not train these models optimally:

min_size: Wastes valuable data from larger dataset
max_size: One model is trained on the static state of the other, causing it to overfit to an old model state
max_size_cycle: May cause overfitting on smaller dataset if dataset sizes are significantly different
sequential: Same problem as max_size but worse

### Pitch

Interleaved mode which calculates how to evenly distribute one dataset with the other. This comes with the caveat of knowing the lengths of each iterable which may not be possible.

e.g.
```python3
iterables = {'a': DataLoader(range(5), batch_size=1),
'b': DataLoader(range(3), batch_size=1)}
combined_loader = CombinedLoader(iterables, 'interleaved')

for batch in combined_loader;
print(batch)
# {'a': tensor([0]), 'b': tensor([0])}
# {'a': tensor([1]), 'b': None}
# {'a': tensor([2]), 'b': tensor([1])}
# {'a': tensor([3]), 'b': None}
# {'a': tensor([4]), 'b': tensor([2])}
```

### Alternatives

_No response_

### Additional context

I don't believe the `__len__` issue would be too damaging as pytorch dataloaders implement a `__len__` method. Obtaining iterable lengths will get stuck on infinite iterables but lengths could otherwise be obtained with:
```python3
sum(1 for _ in iterable)
```

cc @lantiga @borda @tchaton

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate CombinedLoader and inspect how its existing modes determine iteration order and length, then review the PyTorch DataLoader length behavior described in the issue. Done means an interleaved mode produces the shown distribution for unequal-length iterables and handles the stated limitation around unavailable or infinite lengths.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.