apache / apache/beam

FileBasedSink is not creating file paths according to target filesystem

Open
#19,537 1 comment 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

File path generated in _open_writer_ method is not according to target filesystem, because

os.path.join is used and not FileSystems.join.

apache_beam\io\filebasedsink.py extract:

 
```

def _create_temp_dir(self, file_path_prefix):
base_path, last_component = FileSystems.split(file_path_prefix)

if not last_component:
# Trying to re-split the base_path to check if it's a root.
new_base_path,
_ = FileSystems.split(base_path)
if base_path == new_base_path:
raise ValueError('Cannot create
a temporary directory for root path '
'prefix %s. Please specify a file path
prefix with '
'at least two components.' % file_path_prefix)
path_components
= [base_path,
'beam-temp-' + last_component + '-' + uuid.uuid1().hex]
return
FileSystems.join(*path_components)
@check_accessible(['file_path_prefix', 'file_name_suffix'])
def
open_writer(self, init_result, uid):
# A proper suffix is needed for AUTO compression detection.

# We also ensure there will be no collisions with uid and a
# (possibly unsharded) file_path_prefix
and a (possibly empty)
# file_name_suffix.
file_path_prefix = self.file_path_prefix.get()
file_name_suffix
= self.file_name_suffix.get()
suffix = (
'.' + os.path.basename(file_path_prefix) + file_name_suffix)

return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)

```

 

 

This created incompatibilities between, for example, Windows and GCS.
Expected: gs://bucket/beam-temp-result-uuid\\uid.result
Actual: gs://bucket/beam-temp-result-uuid/uid.result

Replacing os.path.join with FileSystems.join fixes the issue

Imported from Jira [BEAM-6821](https://issues.apache.org/jira/browse/BEAM-6821). Original Jira may contain additional context.
Reported by: gkovelman.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.