FileNotFoundError when using SageMaker Debugger with PyTorch Distributed Training on SageMaker

I am using a custom docker image to run distributed training with PyTorch on SageMaker. The training script is taken from https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Segmentation/MaskRCNN. The DLC Image uses `pytorch-training:1.6.0-gpu-py3` as the base image.

Following is the error traceback :

```[1,9]:[2020-10-29 16:13:45.530 algo-2:254 WARNING hook.py:600] Unsupported Distributed Training Strategy Detected. Sagemaker-Debugger will only write from one process. The process with pid: 254 will not be writing any data.
[1,9]:
[1,13]:Traceback (most recent call last):
[1,13]: File "/opt/conda/lib/python3.6/shutil.py", line 550, in move
[1,13]: os.rename(src, real_dst)
[1,13]:FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/output/tensors/events/000000000000/000000000000_worker_0.tfevents.tmp' -> '/opt/ml/output/tensors/events/000000000000/000000000000_worker_0.tfevents'
[1,13]:
[1,13]:During handling of the above exception, another exception occurred:
[1,13]:
[1,13]:Traceback (most recent call last):
[1,13]: File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[1,13]: "__main__", mod_spec)
[1,13]: File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,13]: exec(code, run_globals)
[1,13]: File "/opt/conda/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in
[1,13]: main()
[1,13]: File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
[1,13]: run_command_line(args)
[1,13]: File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
[1,13]: run_path(sys.argv[0], run_name='__main__')
[1,13]: File "/opt/conda/lib/python3.6/runpy.py", line 263, in run_path
[1,13]: pkg_name=pkg_name, script_name=fname)
[1,13]: File "/opt/conda/lib/python3.6/runpy.py", line 96, in _run_module_code
[1,13]: mod_name, mod_spec, pkg_name, script_name)
[1,13]: File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,13]: exec(code, run_globals)
[1,13]: File "train_net.py", line 306, in
[1,13]: main()
[1,13]: File "train_net.py", line 298, in main
[1,13]: model = train(cfg, args)
[1,13]: File "train_net.py", line 165, in train
[1,13]: per_iter_end_callback_fn=per_iter_callback_fn,
[1,13]: File "/root/DeepLearningExamples/PyTorch/Segmentation/MaskRCNN/pytorch/maskrcnn_benchmark/engine/trainer.py", line 78, in do_train
[1,13]: loss_dict = model(images, targets)
[1,13]: File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 724, in _call_impl
[1,13]: result = hook(self, input)
[1,13]: File "/opt/conda/lib/python3.6/site-packages/smdebug/pytorch/hook.py", line 123, in forward_pre_hook
[1,13]: self._close_writers()
[1,13]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/hook.py", line 433, in _close_writers
[1,13]: self.writer.close()
[1,13]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/writer.py", line 201, in close
[1,13]: self._writer.close()
[1,13]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/tfevent/event_file_writer.py", line 125, in close
[1,13]: self._ev_writer.close()
[1,13]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/tfevent/events_writer.py", line 63, in close
[1,13]: self.tfrecord_writer.close()
[1,13]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/tfrecord/record_writer.py", line 81, in close
[1,13]: self._writer.close()
[1,13]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/access_layer/file.py", line 53, in close
[1,13]: shutil.move(self.temp_path, self.path)
[1,13]: File "/opt/conda/lib/python3.6/shutil.py", line 564, in move
[1,13]: copy_function(src, real_dst)
[1,13]: File "/opt/conda/lib/python3.6/shutil.py", line 263, in copy2
[1,13]: copyfile(src, dst, follow_symlinks=follow_symlinks)
[1,13]: File "/opt/conda/lib/python3.6/shutil.py", line 120, in copyfile
[1,13]: with open(src, 'rb') as fsrc:
[1,13]:FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/output/tensors/events/000000000000/000000000000_worker_0.tfevents.tmp'
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 13 in communicator MPI COMMUNICATOR 5 DUP FROM 0
with errorcode 1.
```

@Vikas-kum

Start with smdebug/pytorch/hook.py, smdebug/core/hook.py, smdebug/core/writer.py, and smdebug/core/access_layer/file.py named in the traceback, focusing on writer closure during distributed training. Reproduce the failure with the stated custom Docker image, PyTorch distributed training, and SageMaker Debugger setup; done means the training job no longer aborts with the missing temporary tfevents file.

Written by the indexing model from the issue text.