facebookresearch / facebookresearch/ReAgent

tensorboardX summarywriter file exist error in multiple GPU case

Open
#214 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.7k
Forks
529
PR merge metrics
No merged PRs in 30d

Description

when we run in a one node 4 GPUs case, when our first start, sometimes we get file exist error. usually when we run next time, it get resolve.

but I wonder whether it is a tensorboard issue or ReAgent issue?

```
Traceback (most recent call last):
File "/home/miniconda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/Horizon/ml/rl/workflow/dqn_workflow.py", line 104, in single_process_main
writer = SummaryWriter(log_dir=os.path.join(os.getenv('HOME'), 'tensorboard', os.getenv('JOB_ID')'logs'))
File "/home/miniconda/lib/python3.6/site-packages/tensorboardX/writer.py", line 279, in __init__
self.file_writer = FileWriter(logdir=log_dir, **kwargs)
File "/home/miniconda/lib/python3.6/site-packages/tensorboardX/writer.py", line 201, in __init__
logdir, max_queue, flush_secs, filename_suffix)
File "/home/miniconda/lib/python3.6/site-packages/tensorboardX/event_file_writer.py", line 108, in __init__
directory_check(self._logdir)
File "/home/miniconda/lib/python3.6/site-packages/tensorboardX/record_writer.py", line 43, in directory_check
os.makedirs(path)
File "/home/miniconda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/tensorboard/73e910f9-24b9-41e9-b10e-aedf97ba47a6/logs'
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.