google-deepmind / google-deepmind/scalable_agent

Inconsistent reward clipping

Open
#53 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
160
PR merge metrics
No merged PRs in 30d

Description

Hi there,

I'm looking at the _Optimistic Asymmetric Clipping_ as mentioned in the IMPALA paper, where a clip function f(r) = 0.3 * min(tanh(r), 0) + 5.0 * max(tanh(r), 0) is used. However, I found it is different from the code implementation (https://github.com/deepmind/scalable_agent/blob/master/experiment.py#L367). In the code above, the clip function is f(r) = 5 * 0.3 * min(tanh(r / 5), 0) + 5 * max(tanh(r / 5), 0), which is obviously not consistent with that in the paper.

I plot these two clip functions. Sadly, it seems that both of them doesn't match Figure D.1. in the paper.
![image](https://user-images.githubusercontent.com/57526464/121149298-1ee53a00-c875-11eb-815a-31f8ef928212.png)

Could you please tell which clip function was used in the experiments? And also, any explanations about the discrepancy will be helpful!

Thanks for your reading and I look forward to hearing from you.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.