ContinualAI / ContinualAI/avalanche
feature request: create dataloader for all experiences combined into a single stream without task boundaries
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 321
- PR merge metrics
- No merged PRs in 30d
Description
In the task agnostic setting, we should just get a stream of (x,y) pairs. These may be generated by exctraing N examples from a sequence of empirical distributions (experiences), in a piecewise stationary fashion, but this should be hidden from the user. I wrote some code to do this (see below) but I feel this should be a first-class citizen.
```
def make_avalanche_dataloaders(dataset, ntrain_per_dist, ntest_per_batch, batch_size):
'''Make pytorch dataloaders from avalanche dataset.
ntrain_per_dist: number of training examples from each distribution (experience).
batch_size: how many training examples per batch.
ntest_per_batch: how many test examples per training batch.
'''
train_stream = dataset.train_stream
test_stream = dataset.test_stream
nexperiences = len(train_stream) # num. distinct distributions
nbatches_per_dist = int(ntrain_per_dist / batch_size)
ntest_per_dist = ntest_per_batch * nbatches_per_dist
train_ndx, test_ndx = range(ntrain_per_dist), range(ntest_per_dist)
train_sets = []
test_sets = []
for exp in range(nexperiences):
ds = train_stream[exp].dataset
train_set = torch.utils.data.Subset(ds, train_ndx)
train_sets.append(train_set)
ds = test_stream[exp].dataset
test_set = torch.utils.data.Subset(ds, test_ndx)
test_sets.append(test_set)
train_set = torch.utils.data.ConcatDataset(train_sets)
test_set = torch.utils.data.ConcatDataset(test_sets)
train_dataloader = DataLoader(train_set, batch_size=batch_size, shuffle=False)
test_dataloader = DataLoader(test_set, batch_size=ntest_per_batch, shuffle=False)
return train_dataloader, test_dataloader
```
Contributor guide
Assessment
This issue has not been assessed yet.