google-deepmind / google-deepmind/acme

Is the priority computation correct in prioritized DQN

Open
#205 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4.1k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

Hi,

Cross posting a related issue here
https://github.com/deepmind/reverb/issues/91

After looking at the DQN agent implementation in Acme, I am not sure if the prioritized variant matches the original PER paper.

Specifically, in the original PER, the priority for newly inserted transitions is the maximum of the priority seen so far. However, in Acme, the priority is uniformly set to be 1.0. This deviates from the original PER in that the newly inserted transitions will not be sampled immediately if 1.0 is comparably smaller than the typical TD error in training.

I looked into if it's possible to get the maximum priority in reverb but haven't found a nice way to do so.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.