A question about rl training function

`for action, p, r, b in zip(indices, probs, reward, baseline):
advantage = r - b
avg_advantage += advantage
losses.append(-p.log_prob(action)
* (advantage/len(indices))) # divide by T*B`

I have a question about this piece of code.
If I didn't get it wrong, the variable b here is tensor with gradient enabled, so optimizing tensors in losses will actually both optimize reward by changing policy weights and minimizing the advantage by maximizing baseline. I can't understand why the baseline is optimized here, because as far as I know, the baseline should only be optimized during the training of the critic.
Actually I used this training function in a different summarization task, and I found that the avg_advantage is always dropping.
Thank you very much.

Start from the training function containing the shown zip over indices, probs, reward, and baseline, then inspect how its losses are built and backpropagated. Reproduce the reported decline in avg_advantage and verify whether the observed gradient behavior matches the intended training roles; done means the question is answered with a documented conclusion or a clearly scoped correction.

Written by the indexing model from the issue text.