apache / apache/beam

fileio calls mkdirs with basepath and not dirname(full_file_name)

Open
#21,313 0 comments 0 reactions 0 assignees View on GitHub
bug files io P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When calling `apache_beam.io.fileio.WriteToFiles` with a `file_naming` argument that adds a directory to the path, the current implementation fails to write files if a `mkdirs` or analogous call is needed in the underlying file storage.

Example,

```

apache_beam.io.fileio.WriteToFiles(
path="some/base/dir", sink=..., destination=lambda x: "events",

file_naming=lambda *x: "subdir/file.txt"
)

```

the current fileio implementation will call `mkdirs` with `some/base/dir` instead of `some/base/dir/subdir`.

The bug is currently at https://github.com/apache/beam/blob/67bcf1e16e3fdf68cdea7a4b42b9c003e4b8948c/sdks/python/apache_beam/io/fileio.py#L605.

\==\==

Personally, I would recommend changing the FileSystems interface to have `open` call `mkdirs` in storages that require root parent directory creation.

Imported from Jira [BEAM-13493](https://issues.apache.org/jira/browse/BEAM-13493). Original Jira may contain additional context.
Reported by: txomon.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.