alshedivat / alshedivat/lola

LOLA Policy Gradient Target Computation

Aperta
#8 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Jupyter Notebook
Stelle
156
Fork
38
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Hello, thank you for open-sourcing the code! :-)
The code is really helpful in understanding the papers deeper.

I am interested in LOLA, especially its policy gradient method ([lola/train_pg.py](https://github.com/alshedivat/lola/blob/master/lola/train_pg.py)).
As mentioned in the paper, this implementation shows the actor-critic method.

However, I could not fully understand the target computation code:
`self.target = self.sample_return + self.next_v` ([code](https://github.com/alshedivat/lola/blob/master/lola/networks.py#L155)).
According to the [reference](http://incompleteideas.net/book/bookdraft2017nov5.pdf) (chapter 13, page 274, one-step actor-critic pseudocode), I wonder whether the target computation should use the step reward (i.e., reward at timestep t) instead of the return.

Thank you for your time and consideration!

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.