pytorch / pytorch/rl

[BUG] KeyError for Inverse Transform in ReplayBuffer

Open
#1,163 1 comment 0 reactions 1 assignee View on GitHub

@vmoens is already working on this.

Since May 18, 2023.

bug
Dominant language
Python
Stars
3.6k
Forks
487
Avg merge
1d 1h
Merged PRs (30d)
207

Description

Describe the bug

The inverse transforms on the collected data in the replay buffer are executed on the nested tensordict with keys "_data" and "index". When passing to the inverse transforms data = self._transform.inv(data) we get a KeyError:
KeyError: 'key "(\'next\', \'done\')" not found in TensorDict with keys [\'_data\', \'index\']'.

Changing the line to data.set("_data", self._transform.inv(data.get("_data"))) solves the issue. Following works as well: data = self._transform.inv(data.get("_data"))
However, the second example now does not have the nested tensordict with keys "_data" and "index". I suppose those are important even though the script runs without errors.

@vmoens What do you think? elif case hints that we indeed only want the data and not the nested tensordict with "_data" and "index" keys.

    def extend(self, data: Sequence) -> torch.Tensor:
        """Extends the replay buffer with one or more elements contained in an iterable.

        Args:
            data (iterable): collection of data to be added to the replay
                buffer.

        Returns:
            Indices of the data added to the replay buffer.
        """
        if self._transform is not None and is_tensor_collection(data):
            data = self._transform.inv(data)
        elif self._transform is not None and len(self._transform):
            # Accepts transforms that act on "data" key
            data = self._transform.inv(TensorDict({"data": data}, [])).get("data")
        with self._replay_lock:
            index = self._writer.extend(data)
            self._sampler.extend(index)
        return index

To Reproduce

Steps to reproduce the behavior.

Basically, any example script where you add an inverse transform to the replay buffer like Reward2Go.

Checklist

  • I have checked that there is no similar issue in the repo (required)
  • I have read the documentation (required)
  • I have provided a minimal working example to reproduce the bug (required)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.