Beam IO WriteToText - FileNotFoundError

Hi Team,

We are running a beam pipeline & writing the final output to a file as below. This runs fine most times but fails sometimes with error "FileNotFoundError: [Errno 2] No such file or directory". This has been intermittent and there is no issue with the file location, other jobs using the same location run fine at the same time & this too works fine if we immediately re-try. Please check and advise, I couldn't find any solution/details for this online. Thanks in advance.

 
            | 'write to file' \>\> beam.io.WriteToText(self.output_file,
                                                      header=','.join(output_cols_header),
                                                      shard_name_template=''
                                                      )
 
*****ERROR:*****
[2022-01-17 15:20:43,512] \{bash_operator.py:126} INFO - Traceback (most recent call last):
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "apache_beam/runners/common.py", line 742, in apache_beam.runners.common.PerWindowInvoker.invoke_process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "apache_beam/runners/common.py", line 867, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/iobase.py", line 1148, in process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -     writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py", line 135, in _f
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -     return fnc(self, *args, **kwargs)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 196, in open_writer
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -     return FileBasedSinkWriter(self, writer_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 417, in __init__
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -     self.temp_handle = self.sink.open(temp_shard_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/textio.py", line 405, in open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -     file_handle = super(_TextSink, self).open(temp_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py", line 135, in _f
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -     return fnc(self, *args, **kwargs)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 138, in open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -     return FileSystems.create(temp_path, self.mime_type, self.compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filesystems.py", line 229, in create
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -     return filesystem.create(path, mime_type, compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py", line 168, in create
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -     return self._path_open(path, 'wb', mime_type, compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -   File "/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py", line 143, in _path_open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO -     raw_file = io.open(path, mode)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - FileNotFoundError: [Errno 2] No such file or directory: '/dropbox/weather/prd/kumulus/data/earthsat/daily/2022/01/v2/beam-temp-earthsat_daily_4cast_20220117.csv-0163f65477a911ecb8d80aec9c027dc6/da21a024-9253-4500-9d10-576bdb6a4711.earthsat_daily_4cast_20220117.csv'

Imported from Jira [BEAM-13706](https://issues.apache.org/jira/browse/BEAM-13706). Original Jira may contain additional context.
Reported by: vinodraj.konda.