Reverb adder performance decreases over time

Hi!
I'm using acme library for training the R2D2 agent on Atari games. I'm training on Vertex AI with 128 actor nodes, 1 reverb node, and 1 learner node. After some time of training (~20 hours), the utilisation of CPU cores on actors decreases and the speed of experience collection decreases as well. After some investigation, I've found out that its the function that adds experience to the reverb taking more and more time as the training progresses.

Below you can see the CPU utilisation for all the nodes participating in the training. The green curve corresponds to evaluator (which is almost the same as actor, except for the fact it skips the step of adding experience to reverb).

I'm using:
dm-acme==0.4.0
dm-reverb==0.7.0

I use SequenceAdder for adding the experience, and SampleToInsertRatio for limiting the number of insertions compared to the number of samples on learning. Min size of reverb table is 6250 and max size is 100k.

Start with the SequenceAdder and SampleToInsertRatio configuration described in the report, using the listed dm-reverb 0.7.0 and dm-acme 0.4.0 versions and the long-running 128-actor setup. Done means identifying why insertion slows over time and demonstrating stable experience-collection performance in a comparable training run.

Written by the indexing model from the issue text.