fileio calls mkdirs with basepath and not dirname(full_file_name)
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When calling `apache_beam.io.fileio.WriteToFiles` with a `file_naming` argument that adds a directory to the path, the current implementation fails to write files if a `mkdirs` or analogous call is needed in the underlying file storage.
Example,
```
apache_beam.io.fileio.WriteToFiles(
path="some/base/dir", sink=..., destination=lambda x: "events",
file_naming=lambda *x: "subdir/file.txt"
)
```
the current fileio implementation will call `mkdirs` with `some/base/dir` instead of `some/base/dir/subdir`.
The bug is currently at https://github.com/apache/beam/blob/67bcf1e16e3fdf68cdea7a4b42b9c003e4b8948c/sdks/python/apache_beam/io/fileio.py#L605.
\==\==
Personally, I would recommend changing the FileSystems interface to have `open` call `mkdirs` in storages that require root parent directory creation.
Imported from Jira [BEAM-13493](https://issues.apache.org/jira/browse/BEAM-13493). Original Jira may contain additional context.
Reported by: txomon.
Contributor guide
Assessment
This issue has not been assessed yet.