ContinualAI / ContinualAI/avalanche
`ReplayPlugin` is slow when `num_workers` > 0
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 321
- PR merge metrics
- No merged PRs in 30d
Description
This is an expected issue for the `ReplayPlugin` to be slower when the number of workers is larger than zero, especially when the size of the buffer's dataloader is much smaller than the current task's loader. This happens regardless of the storage policy type. Using `num_workers>0` can be crucial for dataloaders with many augmentations or with large sample sizes.
One potential solution is to initialize both loaders only once before each experience. The max length can be set as the length of the tasks's loader, but instead of initializing the buffer's loader multiple times, we can set a cyclic sampler for the buffer with the same length as the task's loader when initializing it.
P.S.: _I couldn't reproduce the issue that we discussed earlier regarding the interruptions in the class-balanced buffer for the Split-CIFAR100 benchmark (bs=64, mem_size=200). I only encounter that problem in a CIR scenario. I can't say for sure what cause the issue there, but in that particular case, a multi-group buffer has long interruptions at the beginning of the stream._
Contributor guide
Assessment
This issue has not been assessed yet.