google-deepmind / google-deepmind/acme

Reverb Replay Buffers + Vectorized/Batched Environments

Open
#300 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4.1k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

I'm trying to use a reverb replay buffer with a batched environment like 'envpool' where the api returns a batch of experience whenever the either `.reset` or `.step ` is called.

I'm guessing there must be a better way to insert that data into the buffer than to have a writer for each individual environment and iterate over the writers adding their respective batch index of experience experience.

The below is clearly suboptimal and defeats the purpose of using a vectorized environment opposed to many workers executing a single environment.
```
num_envs = 100
envs = make_envs(num_envs)
writer = [client.writer() for _ in range(num_envs)]
obs = envs.reset()
# obs.shape == (100, 3, 86, 86) 100 atari obs

while True:
next_obs, reward, done, info = envs.step(action)
# next_obs.shape == (100, 3, 86, 86)
for i, writer in enumerate(writers):
writer.append({
'obs': obs[i],
.....
}
obs = next_obs
```

If there are any examples of working with batched environments and reverb in the codebase or if anyone could provide some direction, I'd greatly appreciate it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.