apache / apache/beam

FileBasedSink: allow setting temp directory provider per dynamic destination

Open
#20,531 0 comments 0 reactions 0 assignees View on GitHub
bug core hadoop-file-system io java P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Dynamic file destinations allow value-dependent writes in FileBasedSink. When using hadoop file system this means user can write some values to destination at *cluster-A* and some values to destination at *cluster-B*.

Since BEAM-7613 was fixed this works fine until the *moveToOutputFiles* method is called. This method internally calls *FileSystems.rename* which obviously requires that source files (temporary files) and target files (resolved by dynamic destination's function) are on the same cluster. But the temp directory provider can be set only one per file sink.

This could be fixed by adding some kind of *getTempDirectoryProvider* method into dynamic destinations (e.g. into *DefaultFilenamePolicy.Params*).

 

 

Imported from Jira [BEAM-10295](https://issues.apache.org/jira/browse/BEAM-10295). Original Jira may contain additional context.
Reported by: davidak09.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.