~30X slow down with tf-agents 0.19.0

Hi,

I have run circuit_training using an old build from 2022 using the default setting on Ariane.

After updating to the latest build and updating tf-agents 0.19.0, the training drastically slows down with the default setting on Ariane.

I can see the buffer being filled by the collector jobs. However, the code will always get stuck on [learner.run()](https://github.com/google-research/circuit_training/blob/main/circuit_training/learning/train_ppo_lib.py#L369).

This function seemingly comes from [a wrapper class](https://github.com/google-research/circuit_training/blob/main/circuit_training/learning/learner.py) for [tf_agents.train.learner](https://github.com/tensorflow/agents/blob/f854b450184463214908ea9ece5b5aa7fc39f229/tf_agents/train/learner.py).

Is this an expected slowdown by using tf-agents 0.19.0, or is something wrong with my environment?

Reproduce circuit_training with the default Ariane setting and tf-agents 0.19.0, then compare it with the 2022 build. Start at circuit_training/learning/train_ppo_lib.py around learner.run() and the wrapper in circuit_training/learning/learner.py. Done means determining whether the slowdown is reproducible and attributable to tf-agents or the environment, with supporting evidence.

Written by the indexing model from the issue text.