google-deepmind / google-deepmind/reverb

Maintaining episode-level priority while sampling individual transitions from a replay buffer with MinHeap remover

Open
#121 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
793
Forks
113
PR merge metrics
No merged PRs in 30d

Description

Hi,

I have a table containing episodic trajectories. For each episode, I assign a priority. I only want to keep N episodes with the highest priority. For that, I use a MinHeap selector as remover. However, I want to sample individual transitions from all these trajectories, not full episodes. I have tried adding another table that stores individual transitions and sampling from there, but since it keeps a separate remover, the items in both tables do not match when inserting new data. Any ideas how I could achieve this?
On a related note, is it possible to keep a separate priority for sampling (to achieve something like Prioritized Experience Replay) and for removing for the same item?

Any help would be greatly appreciated!

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is named. Start by tracing how the table, MinHeap remover, and sampling behavior interact, then clarify whether the requested outcome is synchronized episode and transition tables, independent sampling and removal priorities, or both; done requires an agreed supported design and tests for the chosen behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.