google / google/dopamine

distribution in softmax_cross_entropy_with_logits

Open
#120 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
10.9k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

The following code uses target distribution in the `softmax_cross_entropy_with_logits`. Is this correct?

https://github.com/google/dopamine/blob/6463bfa8660daf17823825ab884b118d3a57ea4e/dopamine/agents/rainbow/rainbow_agent.py#L259

BTW, I found the code for the projection in c51 is somewhat complicated. I use the following code to compute the projection in Eq.7 in the paper

```python
y = tf.clip_by_value(supports, v_min, v_max)[:, None, :] # [B, 1, N]
target_support = target_support[None, :, None] # [1, N, 1]

y = tf.clip_by_value(1. - tf.abs(y - target_support) / delta_z, 0, 1) # [B, N, N]
y = tf.reduce_sum(y * weights, axis=2) # [B, N]
y = tf.stop_gradient(y)
```
All names except `y` stand for the same meaning as those defined in function `project_distribution`. Do you think they are doing the same thing?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.