google / google/dopamine

Same key used for each state during training of full rainbow

Open
#222 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
10.9k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

Hi, while benchmarking my own implementation of Rainbow I had been using the dopamine full version as a reference, however I noticed that in the get_logits and get_q_values functions in [line 95](https://github.com/google/dopamine/blob/4552f69af4763053d87ee4ce6d3da59ca3232f3c/dopamine/jax/agents/full_rainbow/full_rainbow_agent.py#L95) and [100](https://github.com/google/dopamine/blob/4552f69af4763053d87ee4ce6d3da59ca3232f3c/dopamine/jax/agents/full_rainbow/full_rainbow_agent.py#L100) respectively their is no axis mapped over the key, and later on in the train script only a single key is split for each of the forward passes.

I am just curious, if given the use of Noisy Networks in the full rainbow algorithm, is this implementation detail intentional? As it will lead to the same noisy parameters for each statement and thus a higher bias to the gradient updates. From my understanding the alternative is to split the keys per forward pass and for each input, which will introduce some additional computation. I understand this might additional bias be negligible compared to the computation saved but am interested to hear :)

Thanks for your time.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.