ContinualAI / ContinualAI/avalanche

feature request: create dataloader for all experiences combined into a single stream without task boundaries

Open
#1,305 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
321
PR merge metrics
No merged PRs in 30d

Description

In the task agnostic setting, we should just get a stream of (x,y) pairs. These may be generated by exctraing N examples from a sequence of empirical distributions (experiences), in a piecewise stationary fashion, but this should be hidden from the user. I wrote some code to do this (see below) but I feel this should be a first-class citizen.

```
def make_avalanche_dataloaders(dataset, ntrain_per_dist, ntest_per_batch, batch_size):
'''Make pytorch dataloaders from avalanche dataset.
ntrain_per_dist: number of training examples from each distribution (experience).
batch_size: how many training examples per batch.
ntest_per_batch: how many test examples per training batch.
'''
train_stream = dataset.train_stream
test_stream = dataset.test_stream
nexperiences = len(train_stream) # num. distinct distributions
nbatches_per_dist = int(ntrain_per_dist / batch_size)
ntest_per_dist = ntest_per_batch * nbatches_per_dist
train_ndx, test_ndx = range(ntrain_per_dist), range(ntest_per_dist)

train_sets = []
test_sets = []
for exp in range(nexperiences):
ds = train_stream[exp].dataset
train_set = torch.utils.data.Subset(ds, train_ndx)
train_sets.append(train_set)

ds = test_stream[exp].dataset
test_set = torch.utils.data.Subset(ds, test_ndx)
test_sets.append(test_set)

train_set = torch.utils.data.ConcatDataset(train_sets)
test_set = torch.utils.data.ConcatDataset(test_sets)

train_dataloader = DataLoader(train_set, batch_size=batch_size, shuffle=False)
test_dataloader = DataLoader(test_set, batch_size=ntest_per_batch, shuffle=False)
return train_dataloader, test_dataloader
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.