google-deepmind / google-deepmind/acme
Is the priority computation correct in prioritized DQN
- Dominant language
- Python
- Stars
- 4.1k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
Cross posting a related issue here
https://github.com/deepmind/reverb/issues/91
After looking at the DQN agent implementation in Acme, I am not sure if the prioritized variant matches the original PER paper.
Specifically, in the original PER, the priority for newly inserted transitions is the maximum of the priority seen so far. However, in Acme, the priority is uniformly set to be 1.0. This deviates from the original PER in that the newly inserted transitions will not be sampled immediately if 1.0 is comparably smaller than the typical TD error in training.
I looked into if it's possible to get the maximum priority in reverb but haven't found a nice way to do so.
Contributor guide
Assessment
This issue has not been assessed yet.