google / google/dopamine

setting tf_device = '/gpu:1' mallocs all memory on /gpu:0

Open
#81 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
10.9k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

I ran into a nasty memory usage error, which dopamine is already fairly well structured to take care of. The title is fairly self-explanatory, and running a simultaneous job with tf_device = '/gpu:0' will fail if the gpu:1 job was started first. I haven't done any analysis on how this impacts performance, but setting allow_growth=True in the gpu_options for the session initialized in run_experiment would do the trick. Alternately, setting which gpu to use in the gpu_options (by making it the only one visible) would also work but require Runner to be configured in gin to set the correct gpu, and also needs the handle the nasty cpu-only case, so I don't like that fix due to the complexity of the change.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.