Lightning-AI / Lightning-AI/pytorch-lightning
Interleaved Model for combined_loader
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
Description: A mode for combined_loader which would evenly distribute samples from all iterables.
Motivation: While working with cycle consistency training, two models are updated iteratively on the output of the other model. The current modes for combined_loader do not train these models optimally:
min_size: Wastes valuable data from larger dataset
max_size: One model is trained on the static state of the other, causing it to overfit to an old model state
max_size_cycle: May cause overfitting on smaller dataset if dataset sizes are significantly different
sequential: Same problem as max_size but worse
### Pitch
Interleaved mode which calculates how to evenly distribute one dataset with the other. This comes with the caveat of knowing the lengths of each iterable which may not be possible.
e.g.
```python3
iterables = {'a': DataLoader(range(5), batch_size=1),
'b': DataLoader(range(3), batch_size=1)}
combined_loader = CombinedLoader(iterables, 'interleaved')
for batch in combined_loader;
print(batch)
# {'a': tensor([0]), 'b': tensor([0])}
# {'a': tensor([1]), 'b': None}
# {'a': tensor([2]), 'b': tensor([1])}
# {'a': tensor([3]), 'b': None}
# {'a': tensor([4]), 'b': tensor([2])}
```
### Alternatives
_No response_
### Additional context
I don't believe the `__len__` issue would be too damaging as pytorch dataloaders implement a `__len__` method. Obtaining iterable lengths will get stuck on infinite iterables but lengths could otherwise be obtained with:
```python3
sum(1 for _ in iterable)
```
cc @lantiga @borda @tchaton
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate CombinedLoader and inspect how its existing modes determine iteration order and length, then review the PyTorch DataLoader length behavior described in the issue. Done means an interleaved mode produces the shown distribution for unequal-length iterables and handles the stated limitation around unavailable or infinite lengths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100