ContinualAI / ContinualAI/avalanche

new_instances_benchmark problems

Open
#1,608 2 comments 0 reactions 0 assignees View on GitHub
Feature - Medium Priority
Dominant language
Python
Stars
2.1k
Forks
321
PR merge metrics
No merged PRs in 30d

Description

🐛 **Describe the bug**

One of this problem comprises a bug, which gives the following error when calling classes_in_this_experience on an experience. The same bug already happened for class_incremental_benchmark in combination with benchmark_with_validation_stream (here I am also using benchmark_with_validation_stream).

```
AttributeError: 'DatasetExperience' object has no attribute 'classes_in_this_experience'
```

Otherwise, these are just general things about this method make it unusable. Ideally, when calling this, I would like to do something like this:

```

train_dataset = TinyImagenet(
root=dataset_root, train=True, transform=train_transform
)
test_dataset = TinyImagenet(
root=dataset_root, train=False, transform=eval_transform
)

benchmark = new_instances_benchmark(
train_dataset,
test_dataset,
balance_experiences=True,
shuffle=shuffle,
num_experiences=n_experiences,
seed=seed,
)

benchmark = benchmark_with_validation_stream(
benchmark, validation_size=val_size, shuffle=True
)

```

However, if I want to get it work, I currently have to do something like this

```
train_dataset = TinyImagenet(
root=dataset_root, train=True, transform=train_transform
)
test_dataset = TinyImagenet(
root=dataset_root, train=False, transform=eval_transform
)

train_dataset = ClassificationDataset(
train_dataset,
data_attributes=[
DataAttribute(train_dataset.targets, "targets"),
DataAttribute([0] * len(train_dataset), "targets_task_labels", use_in_getitem=True),
],
)

test_dataset = ClassificationDataset(
test_dataset,
data_attributes=[
DataAttribute(test_dataset.targets, "targets"),
DataAttribute([0] * len(test_dataset), "targets_task_labels", use_in_getitem=True),
],
)

benchmark = new_instances_benchmark(
train_dataset,
test_dataset,
balance_experiences=True,
shuffle=shuffle,
num_experiences=n_experiences,
seed=seed,
)

benchmark = benchmark_with_validation_stream(
benchmark, validation_size=val_size, shuffle=True
)

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.