ContinualAI / ContinualAI/avalanche

`ReplayPlugin` is slow when `num_workers` > 0

Open
#1,348 6 comments 0 reactions 1 assignee Claimed by @HamedHemati View on GitHub
bug
Dominant language
Python
Stars
2.1k
Forks
321
PR merge metrics
No merged PRs in 30d

Description

This is an expected issue for the `ReplayPlugin` to be slower when the number of workers is larger than zero, especially when the size of the buffer's dataloader is much smaller than the current task's loader. This happens regardless of the storage policy type. Using `num_workers>0` can be crucial for dataloaders with many augmentations or with large sample sizes.

One potential solution is to initialize both loaders only once before each experience. The max length can be set as the length of the tasks's loader, but instead of initializing the buffer's loader multiple times, we can set a cyclic sampler for the buffer with the same length as the task's loader when initializing it.

P.S.: _I couldn't reproduce the issue that we discussed earlier regarding the interruptions in the class-balanced buffer for the Split-CIFAR100 benchmark (bs=64, mem_size=200). I only encounter that problem in a CIR scenario. I can't say for sure what cause the issue there, but in that particular case, a multi-group buffer has long interruptions at the beginning of the stream._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.