pytorch / pytorch/rl

[Feature Request] Reset parameter noise at environment reset.

Open
#1,210 1 comment 0 reactions 1 assignee View on GitHub

@vmoens is already working on this.

Since May 30, 2023.

enhancement
Dominant language
Python
Stars
3.6k
Forks
487
Avg merge
1d 1h
Merged PRs (30d)
207

Description

Motivation

Unlike gSDE structured exploration, there is no way for layers with parametric noise NoisyLayer to resample the noise at environment reset. It can only be done after each optimization step, which is not following what is mentioned in the article and introduces bias in training batches.

Solution

For gSDE, there is a special transform specifically for this need gDSENoise, which is made possible because gSDEModel adds its parameters in the rollout TensorDict directly, which is not the case for NoisyLayer. It introduce issues when dealing with multiple workers in parallel since they all rely on the same policy. So I guess the only generic way to handle this situation is to add the action noise as part of the rollout TensorDict, much like what is done for gSDE.

Alternative

Add a hooking mechanism to run generic callbacks at reset of a BaseEnv.

Checklist

  • I have checked that there is no similar issue in the repo (required)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.