facebookresearch / facebookresearch/ReAgent
tensorboardX summarywriter file exist error in multiple GPU case
- Dominant language
- Python
- Stars
- 3.7k
- Forks
- 529
- PR merge metrics
- No merged PRs in 30d
Description
when we run in a one node 4 GPUs case, when our first start, sometimes we get file exist error. usually when we run next time, it get resolve.
but I wonder whether it is a tensorboard issue or ReAgent issue?
```
Traceback (most recent call last):
File "/home/miniconda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/Horizon/ml/rl/workflow/dqn_workflow.py", line 104, in single_process_main
writer = SummaryWriter(log_dir=os.path.join(os.getenv('HOME'), 'tensorboard', os.getenv('JOB_ID')'logs'))
File "/home/miniconda/lib/python3.6/site-packages/tensorboardX/writer.py", line 279, in __init__
self.file_writer = FileWriter(logdir=log_dir, **kwargs)
File "/home/miniconda/lib/python3.6/site-packages/tensorboardX/writer.py", line 201, in __init__
logdir, max_queue, flush_secs, filename_suffix)
File "/home/miniconda/lib/python3.6/site-packages/tensorboardX/event_file_writer.py", line 108, in __init__
directory_check(self._logdir)
File "/home/miniconda/lib/python3.6/site-packages/tensorboardX/record_writer.py", line 43, in directory_check
os.makedirs(path)
File "/home/miniconda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/tensorboard/73e910f9-24b9-41e9-b10e-aedf97ba47a6/logs'
```
Contributor guide
Assessment
This issue has not been assessed yet.