[Feature Request]: Python SDK | File IO | Dynamically update the parent directory.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
We are looking into writing into separate directories using the python apache beam using fileio.WriteToFiles. But it only gives option to write into separate [files](https://github.com/apache/beam/blob/316c969b9c373177986ee0fbc794ed2887244328/sdks/python/apache_beam/io/fileio.py#L678-L679), not in separate parent directories.
We are writing into GCS (Google Cloud Storage) were we really don't have any directory/folder, it is a flat structure. So writing into a new file with different prefix doesn't require any folder creation.
I ran small tests by updating following inside `_MoveTempFilesIntoFinalDestinationFn` method -
```
final_full_path = filesystems.FileSystems.join(
self.path.get(), final_file_name)
```
to
```
final_full_path = final_file_name
```
Provided the whole path through `file_naming=`, it successfully wrote to two separate paths -
```
1. /bigstore/customer-A/dfasfasdfasdfas/...
2. /bigstore/customer-B/jlkjjoijmnyiouuy/...
```
We would like to update have this as part of the feature which would be beneficial while using the flat structure file systems like GCS.
### Issue Priority
Priority: 2
### Issue Component
Component: io-py-files
Contributor guide
Assessment
This issue has not been assessed yet.